org.jsoup.helper

Class HttpConnection

java.lang.Object

org.jsoup.helper.HttpConnection

All Implemented Interfaces:

Connection

public class HttpConnection
extends java.lang.Object
implements Connection

Implementation of Connection.

See Also:

Jsoup.connect(String)

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private static class	HttpConnection.Base<T extends Connection.Base<T>>
static class	HttpConnection.KeyVal
static class	HttpConnection.Request
static class	HttpConnection.Response

Nested classes/interfaces inherited from interface org.jsoup.Connection

Connection.Method

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	CONTENT_ENCODING
static java.lang.String	CONTENT_TYPE
static java.lang.String	DEFAULT_UA Many users would get caught by not setting a user-agent and therefore getting different responses on their desktop vs in jsoup, which would otherwise default to Java.
private static java.lang.String	DefaultUploadType
static java.lang.String	FORM_URL_ENCODED
private static int	HTTP_TEMP_REDIR
private static java.nio.charset.Charset	ISO_8859_1
static java.lang.String	MULTIPART_FORM_DATA
private HttpConnection.Request	req
private Connection.Response	res
private static java.lang.String	USER_AGENT

Constructor Summary

Constructors

Modifier	Constructor and Description
	HttpConnection() Create a new, empty HttpConnection.
(package private)	HttpConnection(HttpConnection.Request copy) Create a new Request by deep-copying an existing Request.
private	HttpConnection(HttpConnection.Request req, HttpConnection.Response res) Create a new Connection that just wraps the provided Request and Response

Method Summary

Modifier and Type	Method and Description
static Connection	connect(java.lang.String url) Create a new Connection, with the request URL specified.
static Connection	connect(java.net.URL url) Create a new Connection, with the request URL specified.
Connection	cookie(java.lang.String name, java.lang.String value) Set a cookie to be sent in the request.
Connection	cookies(java.util.Map<java.lang.String,java.lang.String> cookies) Adds each of the supplied cookies to the request.
java.net.CookieStore	cookieStore() Get the cookie store used by this Connection.
Connection	cookieStore(java.net.CookieStore cookieStore) Provide a custom or pre-filled CookieStore to be used on requests made by this Connection.
Connection	data(java.util.Collection<Connection.KeyVal> data) Adds all of the supplied data to the request data parameters
Connection	data(java.util.Map<java.lang.String,java.lang.String> data) Adds all of the supplied data to the request data parameters
Connection	data(java.lang.String... keyvals) Add one or more request key, val data parameter pairs.
Connection.KeyVal	data(java.lang.String key) Get the data KeyVal for this key, if any
Connection	data(java.lang.String key, java.lang.String value) Add a request data parameter.
Connection	data(java.lang.String key, java.lang.String filename, java.io.InputStream inputStream) Add an input stream as a request data parameter.
Connection	data(java.lang.String key, java.lang.String filename, java.io.InputStream inputStream, java.lang.String contentType) Add an input stream as a request data parameter.
private static java.lang.String	encodeMimeName(java.lang.String val)
Connection.Response	execute() Execute the request.
Connection	followRedirects(boolean followRedirects) Configures the connection to (not) follow server redirects.
Document	get() Execute the request as a GET, and parse the result.
Connection	header(java.lang.String name, java.lang.String value) Set a request header.
Connection	headers(java.util.Map<java.lang.String,java.lang.String> headers) Adds each of the supplied headers to the request.
Connection	ignoreContentType(boolean ignoreContentType) Ignore the document's Content-Type when parsing the response.
Connection	ignoreHttpErrors(boolean ignoreHttpErrors) Configures the connection to not throw exceptions when a HTTP error occurs.
Connection	maxBodySize(int bytes) Set the maximum bytes to read from the (uncompressed) connection into the body, before the connection is closed, and the input truncated (i.e.
Connection	method(Connection.Method method) Set the request method to use, GET or POST.
private static boolean	needsMultipart(Connection.Request req)
Connection	newRequest() Creates a new request, using this Connection as the session-state and to initialize the connection settings (which may then be independently on the returned Connection.Request object).
Connection	parser(Parser parser) Provide an alternate parser to use when parsing the response to a Document.
Document	post() Execute the request as a POST, and parse the result.
Connection	postDataCharset(java.lang.String charset) Sets the default post data character set for x-www-form-urlencoded post data
Connection	proxy(java.net.Proxy proxy) Set the proxy to use for this request.
Connection	proxy(java.lang.String host, int port) Set the HTTP proxy to use for this request.
Connection	referrer(java.lang.String referrer) Set the request referrer (aka "referer") header.
Connection.Request	request() Get the request object associated with this connection
Connection	request(Connection.Request request) Set the connection's request
Connection	requestBody(java.lang.String body) Set a POST (or PUT) request body.
Connection.Response	response() Get the response, once the request has been executed.
Connection	response(Connection.Response response) Set the connection's response
Connection	sslSocketFactory(javax.net.ssl.SSLSocketFactory sslSocketFactory) Set custom SSL socket factory
Connection	timeout(int millis) Set the total request timeout duration.
Connection	url(java.lang.String url) Set the request URL to fetch.
Connection	url(java.net.URL url) Set the request URL to fetch.
Connection	userAgent(java.lang.String userAgent) Set the request user-agent header.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CONTENT_ENCODING

public static final java.lang.String CONTENT_ENCODING

See Also:

Constant Field Values

DEFAULT_UA

public static final java.lang.String DEFAULT_UA

Many users would get caught by not setting a user-agent and therefore getting different responses on their desktop vs in jsoup, which would otherwise default to Java. So by default, use a desktop UA.

See Also:

Constant Field Values

USER_AGENT

private static final java.lang.String USER_AGENT

See Also:

Constant Field Values

CONTENT_TYPE

public static final java.lang.String CONTENT_TYPE

See Also:

Constant Field Values

MULTIPART_FORM_DATA

public static final java.lang.String MULTIPART_FORM_DATA

See Also:

Constant Field Values

FORM_URL_ENCODED

public static final java.lang.String FORM_URL_ENCODED

See Also:

Constant Field Values

HTTP_TEMP_REDIR

private static final int HTTP_TEMP_REDIR

See Also:

Constant Field Values

DefaultUploadType

private static final java.lang.String DefaultUploadType

See Also:

Constant Field Values

ISO_8859_1

private static final java.nio.charset.Charset ISO_8859_1

req

private HttpConnection.Request req

res

@Nullable
private Connection.Response res

Constructor Detail

HttpConnection

public HttpConnection()

Create a new, empty HttpConnection.

HttpConnection

HttpConnection(HttpConnection.Request copy)

Create a new Request by deep-copying an existing Request. Note that the data and body of the original are not copied. All other settings (proxy, parser, cookies, etc) are copied.

Parameters:

copy - the request to copy

HttpConnection

private HttpConnection(HttpConnection.Request req,
                       HttpConnection.Response res)

Create a new Connection that just wraps the provided Request and Response

Method Detail

connect

public static Connection connect(java.lang.String url)

Create a new Connection, with the request URL specified.

Parameters:

url - the URL to fetch from

Returns:

a new Connection object

connect

public static Connection connect(java.net.URL url)

Create a new Connection, with the request URL specified.

Parameters:

url - the URL to fetch from

Returns:

a new Connection object

encodeMimeName

private static java.lang.String encodeMimeName(java.lang.String val)

newRequest

public Connection newRequest()

Description copied from interface: Connection

Creates a new request, using this Connection as the session-state and to initialize the connection settings (which may then be independently on the returned Connection.Request object).

Specified by:

newRequest in interface Connection

Returns:

a new Connection object, with a shared Cookie Store and initialized settings from this Connection and Request

url

public Connection url(java.net.URL url)

Description copied from interface: Connection

Set the request URL to fetch. The protocol must be HTTP or HTTPS.

Specified by:

url in interface Connection

Parameters:

url - URL to connect to

Returns:

this Connection, for chaining

url

public Connection url(java.lang.String url)

Description copied from interface: Connection

Set the request URL to fetch. The protocol must be HTTP or HTTPS.

Specified by:

url in interface Connection

Parameters:

url - URL to connect to

Returns:

this Connection, for chaining

proxy

public Connection proxy(@Nullable
                        java.net.Proxy proxy)

Description copied from interface: Connection

Set the proxy to use for this request. Set to null to disable a previously set proxy.

Specified by:

proxy in interface Connection

Parameters:

proxy - proxy to use

Returns:

this Connection, for chaining

proxy

public Connection proxy(java.lang.String host,
                        int port)

Description copied from interface: Connection

Set the HTTP proxy to use for this request.

Specified by:

proxy in interface Connection

Parameters:

host - the proxy hostname

port - the proxy port

Returns:

this Connection, for chaining

userAgent

public Connection userAgent(java.lang.String userAgent)

Description copied from interface: Connection

Set the request user-agent header.

Specified by:

userAgent in interface Connection

Parameters:

userAgent - user-agent to use

Returns:

this Connection, for chaining

See Also:

DEFAULT_UA

timeout

public Connection timeout(int millis)

Description copied from interface: Connection

Set the total request timeout duration. If a timeout occurs, an SocketTimeoutException will be thrown.

The default timeout is 30 seconds (30,000 millis). A timeout of zero is treated as an infinite timeout.

Note that this timeout specifies the combined maximum duration of the connection time and the time to read the full response.

Specified by:

timeout in interface Connection

Parameters:

millis - number of milliseconds (thousandths of a second) before timing out connects or reads.

Returns:

this Connection, for chaining

See Also:

Connection.maxBodySize(int)

maxBodySize

public Connection maxBodySize(int bytes)

Description copied from interface: Connection

Set the maximum bytes to read from the (uncompressed) connection into the body, before the connection is closed, and the input truncated (i.e. the body content will be trimmed). The default maximum is 2MB. A max size of 0 is treated as an infinite amount (bounded only by your patience and the memory available on your machine).

Specified by:

maxBodySize in interface Connection

Parameters:

bytes - number of bytes to read from the input before truncating

Returns:

this Connection, for chaining

followRedirects

public Connection followRedirects(boolean followRedirects)

Description copied from interface: Connection

Configures the connection to (not) follow server redirects. By default this is true.

Specified by:

followRedirects in interface Connection

Parameters:

followRedirects - true if server redirects should be followed.

Returns:

this Connection, for chaining

referrer

public Connection referrer(java.lang.String referrer)

Description copied from interface: Connection

Set the request referrer (aka "referer") header.

Specified by:

referrer in interface Connection

Parameters:

referrer - referrer to use

Returns:

this Connection, for chaining

method

public Connection method(Connection.Method method)

Description copied from interface: Connection

Set the request method to use, GET or POST. Default is GET.

Specified by:

method in interface Connection

Parameters:

method - HTTP request method

Returns:

this Connection, for chaining

ignoreHttpErrors

public Connection ignoreHttpErrors(boolean ignoreHttpErrors)

Description copied from interface: Connection

Configures the connection to not throw exceptions when a HTTP error occurs. (4xx - 5xx, e.g. 404 or 500). By default this is false; an IOException is thrown if an error is encountered. If set to true, the response is populated with the error body, and the status message will reflect the error.

Specified by:

ignoreHttpErrors in interface Connection

Parameters:

ignoreHttpErrors - - false (default) if HTTP errors should be ignored.

Returns:

this Connection, for chaining

ignoreContentType

public Connection ignoreContentType(boolean ignoreContentType)

Description copied from interface: Connection

Ignore the document's Content-Type when parsing the response. By default this is false, an unrecognised content-type will cause an IOException to be thrown. (This is to prevent producing garbage by attempting to parse a JPEG binary image, for example.) Set to true to force a parse attempt regardless of content type.

Specified by:

ignoreContentType in interface Connection

Parameters:

ignoreContentType - set to true if you would like the content type ignored on parsing the response into a Document.

Returns:

this Connection, for chaining

data

public Connection data(java.lang.String key,
                       java.lang.String value)

Description copied from interface: Connection

Add a request data parameter. Request parameters are sent in the request query string for GETs, and in the request body for POSTs. A request may have multiple values of the same name.

Specified by:

data in interface Connection

Parameters:

key - data key

value - data value

Returns:

this Connection, for chaining

sslSocketFactory

public Connection sslSocketFactory(javax.net.ssl.SSLSocketFactory sslSocketFactory)

Description copied from interface: Connection

Set custom SSL socket factory

Specified by:

sslSocketFactory in interface Connection

Parameters:

sslSocketFactory - custom SSL socket factory

Returns:

this Connection, for chaining

data

public Connection data(java.lang.String key,
                       java.lang.String filename,
                       java.io.InputStream inputStream)

Description copied from interface: Connection

Add an input stream as a request data parameter. For GETs, has no effect, but for POSTS this will upload the input stream.

Specified by:

data in interface Connection

Parameters:

key - data key (form item name)

filename - the name of the file to present to the remove server. Typically just the name, not path, component.

inputStream - the input stream to upload, that you probably obtained from a FileInputStream. You must close the InputStream in a finally block.

Returns:

this Connections, for chaining

See Also:

if you want to set the uploaded file's mimetype.

data

public Connection data(java.lang.String key,
                       java.lang.String filename,
                       java.io.InputStream inputStream,
                       java.lang.String contentType)

Description copied from interface: Connection

Add an input stream as a request data parameter. For GETs, has no effect, but for POSTS this will upload the input stream.

Specified by:

data in interface Connection

Parameters:

key - data key (form item name)

filename - the name of the file to present to the remove server. Typically just the name, not path, component.

inputStream - the input stream to upload, that you probably obtained from a FileInputStream.

contentType - the Content Type (aka mimetype) to specify for this file. You must close the InputStream in a finally block.

Returns:

this Connections, for chaining

data

public Connection data(java.util.Map<java.lang.String,java.lang.String> data)

Description copied from interface: Connection

Adds all of the supplied data to the request data parameters

Specified by:

data in interface Connection

Parameters:

data - map of data parameters

Returns:

this Connection, for chaining

data

public Connection data(java.lang.String... keyvals)

Description copied from interface: Connection

Add one or more request key, val data parameter pairs.

Multiple parameters may be set at once, e.g.: .data("name", "jsoup", "language", "Java", "language", "English"); creates a query string like: ?name=jsoup&language=Java&language=English

For GET requests, data parameters will be sent on the request query string. For POST (and other methods that contain a body), they will be sent as body form parameters, unless the body is explicitly set by Connection.requestBody(String), in which case they will be query string parameters.

Specified by:

data in interface Connection

Parameters:

keyvals - a set of key value pairs.

Returns:

this Connection, for chaining

data

public Connection data(java.util.Collection<Connection.KeyVal> data)

Description copied from interface: Connection

Adds all of the supplied data to the request data parameters

Specified by:

data in interface Connection

Parameters:

data - collection of data parameters

Returns:

this Connection, for chaining

data

public Connection.KeyVal data(java.lang.String key)

Description copied from interface: Connection

Get the data KeyVal for this key, if any

Specified by:

data in interface Connection

Parameters:

key - the data key

Returns:

null if not set

requestBody

public Connection requestBody(java.lang.String body)

Description copied from interface: Connection

Set a POST (or PUT) request body. Useful when a server expects a plain request body, not a set for URL encoded form key/value pairs. E.g.:

Jsoup.connect(url)
 .requestBody(json)
 .header("Content-Type", "application/json")
 .post();

If any data key/vals are supplied, they will be sent as URL query params.

Specified by:

requestBody in interface Connection

Returns:

this Request, for chaining

header

public Connection header(java.lang.String name,
                         java.lang.String value)

Description copied from interface: Connection

Set a request header.

Specified by:

header in interface Connection

Parameters:

name - header name

value - header value

Returns:

this Connection, for chaining

See Also:

Connection.Base.headers()

headers

public Connection headers(java.util.Map<java.lang.String,java.lang.String> headers)

Description copied from interface: Connection

Adds each of the supplied headers to the request.

Specified by:

headers in interface Connection

Parameters:

headers - map of headers name -> value pairs

Returns:

this Connection, for chaining

See Also:

Connection.Base.headers()

cookie

public Connection cookie(java.lang.String name,
                         java.lang.String value)

Description copied from interface: Connection

Set a cookie to be sent in the request.

Specified by:

cookie in interface Connection

Parameters:

name - name of cookie

value - value of cookie

Returns:

this Connection, for chaining

cookies

public Connection cookies(java.util.Map<java.lang.String,java.lang.String> cookies)

Description copied from interface: Connection

Adds each of the supplied cookies to the request.

Specified by:

cookies in interface Connection

Parameters:

cookies - map of cookie name -> value pairs

Returns:

this Connection, for chaining

cookieStore

public Connection cookieStore(java.net.CookieStore cookieStore)

Description copied from interface: Connection

Provide a custom or pre-filled CookieStore to be used on requests made by this Connection.

Specified by:

cookieStore in interface Connection

Parameters:

cookieStore - a cookie store to use for subsequent requests

Returns:

this Connection, for chaining

cookieStore

public java.net.CookieStore cookieStore()

Description copied from interface: Connection

Get the cookie store used by this Connection.

Specified by:

cookieStore in interface Connection

Returns:

the cookie store

parser

public Connection parser(Parser parser)

Description copied from interface: Connection

Provide an alternate parser to use when parsing the response to a Document. If not set, defaults to the HTML parser, unless the response content-type is XML, in which case the XML parser is used.

Specified by:

parser in interface Connection

Parameters:

parser - alternate parser

Returns:

this Connection, for chaining

get

public Document get()
             throws java.io.IOException

Description copied from interface: Connection

Execute the request as a GET, and parse the result.

Specified by:

get in interface Connection

Returns:

parsed Document

Throws:

java.net.MalformedURLException - if the request URL is not a HTTP or HTTPS URL, or is otherwise malformed

HttpStatusException - if the response is not OK and HTTP response errors are not ignored

UnsupportedMimeTypeException - if the response mime type is not supported and those errors are not ignored

java.net.SocketTimeoutException - if the connection times out

java.io.IOException - on error

post

public Document post()
              throws java.io.IOException

Description copied from interface: Connection

Execute the request as a POST, and parse the result.

Specified by:

post in interface Connection

Returns:

parsed Document

Throws:

java.net.MalformedURLException - if the request URL is not a HTTP or HTTPS URL, or is otherwise malformed

HttpStatusException - if the response is not OK and HTTP response errors are not ignored

UnsupportedMimeTypeException - if the response mime type is not supported and those errors are not ignored

java.net.SocketTimeoutException - if the connection times out

java.io.IOException - on error

execute

public Connection.Response execute()
                            throws java.io.IOException

Description copied from interface: Connection

Execute the request.

Specified by:

execute in interface Connection

Returns:

a response object

Throws:

java.net.MalformedURLException - if the request URL is not a HTTP or HTTPS URL, or is otherwise malformed

HttpStatusException - if the response is not OK and HTTP response errors are not ignored

UnsupportedMimeTypeException - if the response mime type is not supported and those errors are not ignored

java.net.SocketTimeoutException - if the connection times out

java.io.IOException - on error

request

public Connection.Request request()

Description copied from interface: Connection

Get the request object associated with this connection

Specified by:

request in interface Connection

Returns:

request

request

public Connection request(Connection.Request request)

Description copied from interface: Connection

Set the connection's request

Specified by:

request in interface Connection

Parameters:

request - new request object

Returns:

this Connection, for chaining

response

public Connection.Response response()

Description copied from interface: Connection

Get the response, once the request has been executed.

Specified by:

response in interface Connection

Returns:

response

response

public Connection response(Connection.Response response)

Description copied from interface: Connection

Set the connection's response

Specified by:

response in interface Connection

Parameters:

response - new response

Returns:

this Connection, for chaining

postDataCharset

public Connection postDataCharset(java.lang.String charset)

Description copied from interface: Connection

Sets the default post data character set for x-www-form-urlencoded post data

Specified by:

postDataCharset in interface Connection

Parameters:

charset - character set to encode post data

Returns:

this Connection, for chaining

needsMultipart

private static boolean needsMultipart(Connection.Request req)