Call Bulk API using HTTPEndpoint #37
Conversation
|
|
thuva9872
force-pushed
the
bulkapi-v2-new
branch
from
7777a33 to
910cfe5
Compare
| @@ -30,5 +30,109 @@ | |||
| <property name="assignmentRuleId" expression="$func:assignmentRuleId"/> | |||
| <property name="externalIdFieldName" expression="$func:externalIdFieldName"/> | |||
| <class name="org.wso2.carbon.esb.connector.operations.CreateJob" /> | |||
| <property name="uri.var.url" expression="get-property('createJobUrl')"/> | |||
| <property name="uri.var.accessToken" expression="get-property('accessToken')"/> | |||
| <filter source="boolean($ctx:uri.var.accessToken)" regex="true"> | |||
There was a problem hiding this comment.
Shall we use a common sequence and avoid these repetitive code blocks? Refer https://github.com/wso2-extensions/esb-connector-salesforcebulk/blob/master/src/main/resources/salesforceBulk-config/callWithRetry.xml to get an idea
| public static final String GET_UNPROCESSED_RESULTS_URL = "getUnprocessedResultsUrl"; | ||
| public static final String UPLOAD_JOB_DATA_URL = "uploadJobUrl"; | ||
| public static final String TOKEN_URL = "tokenUrl"; | ||
| public static final String PAYLOAD = "payload"; |
There was a problem hiding this comment.
payload is too simple, it can easily get conflict. Shall we put some name with a prefix. like sf__bulk_payload, let's add the prefix to all such properties we are setting to the messageContext.
ex: SF_OAUTH_CONFIG_NAME = "name"; should be changed to SF_OAUTH_CONFIG_NAME = "sf_bulk_name";
| </endpoint> | ||
| </call> | ||
| <property expression="$axis2:HTTP_SC" name="httpCode" scope="default" type="STRING"/> | ||
| <filter regex="40(?:1|3|7|9)|44(?:0|1)|45(?:1|9)|499" source="$ctx:httpCode"> |
There was a problem hiding this comment.
Can we use $axis2:HTTP_SC at the filter without setting it to a new property?
| .setPrettyPrinting() | ||
| .create(); | ||
| String jsonString = gson.toJson(jsonObjectList); | ||
| jsonString = jsonString.replace("\\\\", "\\"); |
There was a problem hiding this comment.
Can you please explain the purpose of this jsonString = jsonString.replace("\\", "\");
There was a problem hiding this comment.
Earlier, it was escaping the escape characters again and transformed '\n' to '\n' like these. It is not needed anymore. I'll remove it.
| </call> | ||
| </else> | ||
| </filter> | ||
| <filter regex="200" source="$axis2:HTTP_SC"> |
There was a problem hiding this comment.
What will happen when outputtype is json and there is a failure?
There was a problem hiding this comment.
By default, saleforce bulk api sends failure messages in application/json format.
Purpose
The Bulk API requests were sent using java requests which couldn't handle high load. So, we needed an implementation that can handle high load.
Goals
Send Bulk API through the synapse HTTPEndpoint which can handle high load.
Related PRs
Documentation: wso2/docs-apim#7213
Test environment
OS : macOS 12.2
JDK : OpenJDK 11.0.19