When invoking any REST web services, a lot of AL code mostly looks like this:
procedure CallRESTFoo()
var
Client: HttpClient;
Response: HttpResponseMessage;
Body: Text;
Json: JsonObject;
begin
Client.Get('https://foo.bar/', Response);
Response.Content.ReadAs(Body);
Json.ReadFrom(Body);
// Process JSON body of the response...
end;
Of course, there are more things there, like headers or perhaps calling HTTP POST (or another method) instead of GET, but when you strip it down to the bones, the chunk above is what remains.
Today’s post is a follow up for my HttpClient Patterns live session on http://vjeko.live/ and as I promised, I am providing the text-only version for those who prefer reading to watching.
You can follow all code examples on the public GitHub repo that I used during my live session: https://github.com/vjekob/HttpClientPatterns/
Naive invocations
As I mentioned in the intro, the code example I provided is bare bones for a typical REST invocation. Something more real-life may look as the src/MetaWeather.Codeunit.al file shows:
<code>procedure GetForecast(WOEID: Integer; var TempForecast: Record "Weather Forecast" temporary)
var
Client: HttpClient;
Response: HttpResponseMessage;
Content: Text;
Results: JsonObject;
ConsolidatedWeather: JsonArray;
JToken: JsonToken;
WeatherObject: JsonObject;
begin
Client.Get(GetForecastUrl(WOEID), Response);
TempForecast.Reset();
TempForecast.DeleteAll();
Response.Content.ReadAs(Content);
Results.ReadFrom(Content);
ConsolidatedWeather := GetToken(Results, 'consolidated_weather').AsArray();
foreach JToken in ConsolidatedWeather do begin
WeatherObject := JToken.AsObject();
TempForecast.ID := GetTokenValue(WeatherObject, 'id').AsBigInteger();
TempForecast.Date := GetTokenValue(WeatherObject, 'applicable_date').AsDate();
TempForecast.Description := GetTokenValue(WeatherObject, 'weather_state_name').AsText();
TempForecast."Min. Temperature" := GetTokenValue(WeatherObject, 'min_temp').AsDecimal();
TempForecast."Max. Temperature" := GetTokenValue(WeatherObject, 'max_temp').AsDecimal();
TempForecast.Insert();
end;
end;(This example is included in the main branch)
I called this a “naive way” of invoking REST. It’s naive because it just flatly calls the HTTP endpoint without any error or response handling at all.
One argument for this kind of invocation that I heard is almost valid: when a call fails with an error, the runtime error will give you enough context that you’ll know how to handle, just like calling Rec.Insert or another database operation without explicit error handling.
True, as I said, it’s almost valid. As far as any Rec.Insert or Rec.Modify can be as valid as if not Rec.Insert then or if not Rec.Modify then you could argue that a plain Client.Get or Client.Post can be equally as valid as if not Client.Get then or if not Client.Post then. You would almost be right.
The big difference between a database operation and a HTTP invocation is that when a database operation fails, it always fails in the same way. HTTP operations can fail on two different layers: transport layer, or HTTP layer.
Transport layer in this case is TCP/IP itself. This is when invocation fails before you even reach the server. If you cannot reach the host for any reason (network down, timeout, whatever) then you get a transport layer error. And that’s the only one you’ll capture with the if part of your “error handling”, or the only error that will propagate to the UI if you don’t inspect the result of Client method with if (or capturing the result into a boolean).
Another layer that things can fail is HTTP. This is when your invocation reached the server, but then the server either rejected your call or failed during executing it. It has to do with HTTP Status Codes.
I’ve heard arguments that as long as you invoke a known API with a tested piece of code, and transport succeeds, then your call should succeed, just like when you call a known method of a known .NET API – it’s deterministic.
Yeah, it is, in theory. In great deal of cases it will work. In theory, if you have the same version of API on the other end, the owner of that API should not change that version, and if they want any updates, they will deploy a new version with a new sets of endpoints. But that’s theory, and it all depends on how exactly the owner of that API maintains it. To make the long story short, if you trust them to never ever hotfix anything on an existing endpoint may be naive. That’s why I call that “pattern” I introduced – naive.
Don’t call your REST endpoints like that.
Split business logic from HTTP
On top of it, there is another antipattern in my second code example: it combines the business logic with the HTTP call. Don’t do this either. I have a better example in my step-1 branch on the repo, where I split the code between HTTP part and business logic part:
local procedure JsonToForecast(Results: JsonObject; var TempForecast: Record "Weather Forecast" temporary)
var
ConsolidatedWeather: JsonArray;
JToken: JsonToken;
WeatherObject: JsonObject;
begin
TempForecast.Reset();
TempForecast.DeleteAll();
ConsolidatedWeather := GetToken(Results, 'consolidated_weather').AsArray();
foreach JToken in ConsolidatedWeather do begin
WeatherObject := JToken.AsObject();
TempForecast.ID := GetTokenValue(WeatherObject, 'id').AsBigInteger();
TempForecast.Date := GetTokenValue(WeatherObject, 'applicable_date').AsDate();
TempForecast.Description := GetTokenValue(WeatherObject, 'weather_state_name').AsText();
TempForecast."Min. Temperature" := GetTokenValue(WeatherObject, 'min_temp').AsDecimal();
TempForecast."Max. Temperature" := GetTokenValue(WeatherObject, 'max_temp').AsDecimal();
TempForecast.Insert();
end;
end;
procedure GetLocations(Search: Text; var TempLocation: Record "Weather Location" temporary)
var
Client: HttpClient;
Response: HttpResponseMessage;
Content: Text;
LocationsArray: JsonArray;
begin
Client.Get(GetSearchUrl(Search), Response);
Response.Content.ReadAs(Content);
LocationsArray.ReadFrom(Content);
JsonToLocation(LocationsArray, TempLocation);
end;Better, but still naive as far as HTTP part goes. However, if anything, you should always split your code to separate the business logic part, the parsing of the response part of it from the HTTP invocation themselves. It will be far easier to maintain, to test, and it generally play better with separation of concerns and the single-responsibility principle.
Before I move on to the next improvement, I’ll just mention that the step-2 branch in my repo simply illustrates the problem with failing transport and succeeding transport but still failing on HTTP. When you run that branch, it fails on transport level and shows a nice error message (the one that you may argue it’s okay if your users saw). Now comment line 4 in the MetaWeather.Codeunit.al file and uncomment line 5, and you get an error that you never want your users to see. Transport succeeded, but the server rejected your request, and you attempted to parse it.
Always inspect the status code
Take a look at what the step-3 branch brings to the table:
if (not Client.Get(GetSearchUrl(Search), Response)) or (not Response.IsSuccessStatusCode()) then begin
// TODO: Log the error, handle it, do something about it here
Error('Something went wrong: %1', GetLastErrorText);
end;This gives you the minimum boilerplate you need to inspect what happened. It’s far from good, but a bit better. The comment TODO line is just to indicate the place where you need to inspect the error. This can look vastly different depending on what exactly you want to do here, or what REST you are talking to, and that’s why I didn’t go in-depth in this.
The important takeaway here is that this block checks for both ways a REST call from AL can fail: the first part (not Client.Get(...)) cares about the transport layer, and the second part (not Response.IsSuccessStatusCode()) cares about the HTTP layer.
There is another – totally AL and BC specific part that I didn’t explicitly address in my live session – but I will address it later in this post – a failure on environment configuration. Bear with me.
In any case, the above block handles the situation much better, and at minimum your invocation should do those two tests and then handle the situation whatever way you want to handle it.
Filter method
I don’t know anymore why I call this approach the filter method approach – way back when I first played with HttpClient years ago I named it that way and I’ve used that term in all my trainings, so I keep using it nowadays.
What is a filter method? It’s a method you call when you want to “filter” results of another method invocation and do some typical processing that you would otherwise have to do every single time.
Scroll a bit back and take another look at that boilerplate example I gave last. It’s okay, but you have to do it every time for every single invocation, and you can have hundreds of different places in code where you need to do the same kind of checking. That’s where the filter method kicks in – it does the handling, and it tells you whether everything was fine, or something failed.
Here’s my proposal:
local procedure IsSuccessfulRequest(TransportOK: Boolean; Response: HttpResponseMessage): Boolean
begin
if TransportOK and Response.IsSuccessStatusCode() then
exit(true);
// TODO: Log the error, handle it, do something about it here
Error('Something went wrong: %1', GetLastErrorText);
end;
procedure GetLocations(Search: Text; var TempLocation: Record "Weather Location" temporary)
var
Client: HttpClient;
Response: HttpResponseMessage;
Content: Text;
LocationsArray: JsonArray;
begin
if not IsSuccessfulRequest(Client.Get(GetSearchUrl(Search), Response), Response) then
exit;
Response.Content.ReadAs(Content);
LocationsArray.ReadFrom(Content);
JsonToLocation(LocationsArray, TempLocation);
end;Now, you call your HTTP endpoint directly from your HTTP method, but you filter it through the same method that takes care of everything else. While still not the best thing in the universe, it’s much better than the previous example.
The filter method itself can do different things. One more comprehensive pattern that I always present in my Azure Functions trainings is this:
local procedure IsSuccessfulRequest(TransportOK: Boolean; Response: HttpResponseMessage): Boolean
begin
case true of
TransportOK and Response.IsSuccessStatusCode():
begin
// Successful response
exit(true);
end;
TransportOK:
begin
// Transport ok, but some kind of server error
// You must read the response here and inspect the status code
// TODO: handle it
end;
Response.IsBlockedByEnvironment():
begin
// Policy of user action blocked the request
end;
else
begin
// Transport problem
end;
end
end;This one takes care of all possible ways a call can fail and handles them appropriately. Yes, the beef is still missing, but the bare bones structure of what you have to do is here.
I didn’t present this in my live session, because I really wanted to keep it simple and get to the final point sooner, but here I can spare a minute to address that last way that an HttpClient call can fail in AL: <a href="https://docs.microsoft.com/en-us/dynamics365/business-central/dev-itpro/developer/methods-auto/httpresponsemessage/httpresponsemessage-isblockedbyenvironment-method" target="_blank" rel="noreferrer noopener">IsBlockedByEnvironment</a>.
While most of Http stack in AL comes directly from System.Net.HttpClient of .NET, this method is entirely AL. When there is a system policy or user action preventing an invocation of an HTTP endpoint, your client call will return false, and IsBlockedByEnvironment will return true. That’s how you know that it’s the configuration (or user choice) that made the call fail, and then you can take steps (like instructing the user about how to reconfigure the system) to avoid the problem in the future. But this is neither HTTP nor transport problem, it’s a BC thing (not a problem, really, just a “thing”).
Let’s move on.
Handling HTTP centrally
My step-5 branch shows an even better approach: handling all HTTP invocation centrally. As I mentioned in the live session, the example I gave there is much less comprehensive than what Waldo did in his <a href="https://github.com/waldo1001/waldo.restapp" target="_blank" rel="noreferrer noopener">https://github.com/waldo1001/waldo.restapp</a> repo, but serves the purpose of explaining what you can do. And what you should do.
Instead of using HttpClient directly, you should use a codeunit that wraps it. That codeunit can then take care of all the complexities you may encounter otherwise (like handling errors), but add anything else that you want every time: logging, checking security, setting up security, events, whatnot. Your choice entirely.
Interfaces
The last step of my improvement comes from my step-6 branch. There, I have an approach to HTTP very similar to method codeunits pattern advocated by Gary Winter, and evangelized by him and Waldo vigorously over the past – well, close to a decade perhaps.
Just like you would have one codeunit per a “method” (an encapsulated chunk of business logic) you may prefer having one codeunit per a REST method (an encapsulated chunk of business logic related to invoking one specific REST endpoint with one specific HTTP method).
This is what I proposed:
interface IHttpRequest
{
procedure Execute(var Response: HttpResponseMessage): Boolean;
}And then:
codeunit 50102 "MetaWeather Search" implements IHttpRequest
{
var
SearchUrl: Label 'https://www.metaweather.com/api/location/search/?query=%1', Locked = true;
LocationsArray: JsonArray;
SearchTerm: Text;
procedure Search(SearchTerm2: Text)
begin
SearchTerm := SearchTerm2;
end;
procedure Execute(var Response: HttpResponseMessage): Boolean
var
Client: HttpClient;
Content: Text;
begin
if not Client.Get(StrSubstNo(SearchUrl, SearchTerm), Response) then
exit(false);
Response.Content.ReadAs(Content);
LocationsArray.ReadFrom(Content);
exit(true);
end;
procedure ReadSearchResults(var TempLocation: Record "Weather Location")
var
JToken: JsonToken;
LocationObject: JsonObject;
Json: Codeunit "Json Helper";
begin
TempLocation.Reset();
TempLocation.DeleteAll();
foreach JToken in LocationsArray do begin
LocationObject := JToken.AsObject();
TempLocation.WOEID := Json.GetTokenValue(LocationObject, 'woeid').AsInteger();
TempLocation."Location Type" := Json.GetTokenValue(LocationObject, 'location_type').AsText();
TempLocation.Location := Json.GetTokenValue(LocationObject, 'title').AsText();
TempLocation.Insert();
end;
end;
}… and then:
codeunit 50103 "MetaWeather Forecast" implements IHttpRequest
{
var
ForecastUrl: Label 'https://www.metaweather.com/api/location/%1/', Locked = true;
WOEID: Integer;
Results: JsonObject;
procedure ForecastFor(WOEID2: Integer)
begin
WOEID := WOEID2;
end;
procedure Execute(var Response: HttpResponseMessage): Boolean;
var
Client: HttpClient;
Content: Text;
begin
if not Client.Get(StrSubstNo(ForecastUrl, WOEID), Response) then
exit(false);
Response.Content.ReadAs(Content);
Results.ReadFrom(Content);
exit(true);
end;
procedure ReadForecast(var TempForecast: Record "Weather Forecast" temporary)
var
ConsolidatedWeather: JsonArray;
JToken: JsonToken;
WeatherObject: JsonObject;
Json: Codeunit "Json Helper";
begin
TempForecast.Reset();
TempForecast.DeleteAll();
ConsolidatedWeather := Json.GetToken(Results, 'consolidated_weather').AsArray();
foreach JToken in ConsolidatedWeather do begin
WeatherObject := JToken.AsObject();
TempForecast.ID := Json.GetTokenValue(WeatherObject, 'id').AsBigInteger();
TempForecast.Date := Json.GetTokenValue(WeatherObject, 'applicable_date').AsDate();
TempForecast.Description := Json.GetTokenValue(WeatherObject, 'weather_state_name').AsText();
TempForecast."Min. Temperature" := Json.GetTokenValue(WeatherObject, 'min_temp').AsDecimal();
TempForecast."Max. Temperature" := Json.GetTokenValue(WeatherObject, 'max_temp').AsDecimal();
TempForecast.Insert();
end;
end;
}Here, there are two codeunits that both implement the same interface. You can consume them like this:
codeunit 50101 "Http Management"
{
local procedure IsSuccessfulRequest(TransportOK: Boolean; Response: HttpResponseMessage) Success: Boolean
begin
Success := TransportOK and Response.IsSuccessStatusCode();
// TODO: log the details about the request:
// - who
// - when
// - what URL
// - success status
// - error
// - ...
end;
procedure Execute(Request: Interface IHttpRequest) Success: Boolean
var
Response: HttpResponseMessage;
begin
// TODO: perform pre-invocation checks
// - validate permissions
// - validate URL
// - ...
Success := IsSuccessfulRequest(Request.Execute(Response), Response);
end;
}The biggest benefit here is testability. When you structure your code this way, and use interfaces to represent code that talks to REST, you can make your AL code testable without having to depend on the REST service being present during testing. The last thing you want is your tests to fail because whatever reason not related to the code you are testing (like – environment may be causing your REST calls to fail in your test environment, but it always works in your production environment).
That’s where I ended my live session, and that’s where I end my post today.
Next steps
The next steps from here are implementing the testability around interfaces, and – perhaps – talk a bit deeper about HTTP error handling. These are topics for my other sessions. I’ll cover the testability part in depth in my testability webinar that I announced last week (but no fixed schedule is available yet), and the deeper HTTP error handling is something I may cover in the future here and on Vjeko.live.
Pingback: A couple of ideas for HttpClient - Vjeko.com - Dynamics 365 Business Central/NAV User Group - Dynamics User Group