iOS: Update from {N} 8.0 to 8.1 creates toString from null error

[teha-at]

I updated my App from NativeScript 8.0 to 8.1 and on iOS I wasn't able to load data from the sqlite db. The console give me

CONSOLE ERROR: ERROR TypeError: Cannot read property 'toString' of null

I don't no why the error appears, but in this line he occurred
https://github.com/NathanaelA/nativescript-sqlite/blob/master/src/sqlite-internal.ios.js#L713
The NSString.stringWithUTF8String method returns null.

I have not yet been able to identify the query that triggers the error. How can I decode the statement to get the query? Should there be a check if the result null?

I use the version 2.8.4 from nativescript-sqlite.

[NathanaelA]

Hmm, that is interesting.

1. Are you sure it is when you upgraded from 8.0 to 8.1?
2. Which iOS version exactly?
3. Are you seeing this every single run, or frequently or only once so far?

At this point the query doesn't "exist" as we are looping thru the result sets. So if you want to know which query, I would recommend you console log it out before you call the command to get the result set...

Now what is very interesting about this issue is:
const resultType = sqlite3_column_type(statement, column); is supposed to tell us what type of data we are to expect and then we called the code to get that type.

So either:

1. You have found a runtime bug in NativeScript 8.1
2. Something in this version's of iOS SQLite or NSString conversion is broken. (So I need to know exactly which iOS version)
3. You found some weird corner case, as it is possible that your underlying data is null and somehow got tagged as string.

[Ed-Simmons]

Hi @NathanaelA, I'm experiencing this issue too, so here's some more info about the sql being executed:

[ERROR] TypeError: Cannot read property 'toString' of null
[ERROR] {SQL} SELECT * FROM migrations WHERE key = "m2020_10_09_084300_create_users_table"

Initially I thought that it was only occurring when creating the tables (which, weirdly, were successfully created despite the error messages), but it's actually happening on every query. Here's another example:

[ERROR] TypeError: Cannot read property 'toString' of null
[ERROR] {SQL} SELECT * FROM questions WHERE is_field_question = "true" AND report_type_id = "8e11630c-5d5e-45f6-ad26-e8b2df78aa10" AND section IN ("1", "2", "5", "7", "8", "120") AND question_option_uuid IS NULL

I can confirm that this only occurs on @nativescript/ios": "8.1.0" (I was actually running 8.1.0-alpha.7), downgrading to @nativescript/ios": "8.0.0" resolved the issue.

I ran all my tests on iOS 14.4

Hope that's helpful? Let me know if I can provide anything more useful!

[NathanaelA]

@Ed-Simmons - Can you tell me is there any null values in the data on 8.0.0 for either of those sql queries when they fail. Narrowing down why/when this occurs would be very helpful. I'll also run my test suite on 8.1.0 just to see if any of these queries break,

[Ed-Simmons]

@NathanaelA - Indeed there are some null values. If we take the second query I mentioned as an example, there are two nullable columns:

The error is being caught in the callback of the all() method, not sure if that helps narrow it down:

  public execute(sql: string, parameters?: string[]): Promise<any> {
    return this.sqlite.all(sql, parameters).then((records: object[][]) => {
      return records;
    }, (error: any) => {
      Database.debug(`[ERROR] ${error}`);
      Database.debug(`[ERROR] {SQL} ${sql}`);
      if (parameters) {
        Database.debug(`[ERROR] {PARAMS} ${parameters}`);
      }
      return error;
    });
  }

[NathanaelA]

Is it possible I can get a copy of your db or even a sample that fails. (I'm willing to sign a NDA)

I updated my test framework to include a record with nulls (https://github.com/NathanaelA/nativescript-sqlite/blob/master/demo/app/main-page.js#L185) in each type of field, had it loop thru all the tests 20 times in a row and not a single failure using NS 8.1 ios runtime.

So either:

1. Something with the OS (which version of iOS?)
2. Real device or emulator? (I was using a real device)
3. Some corner case that I can't seem to duplicate

You can try my test framework by downloading the demo in this repo, and then using the name "test" and it should run thru all the tests testing all the functionality of the sqlite framework.

[fjohnston]

Also experiencing the same issue:

Tested iOS versions: 13.7 (emulator), 14.5 (emulator), 14.7.1 (iPad mini 5th gen)

  "dependencies": {
    "@nativescript/background-http": "^5.0.2",
    "@nativescript/core": "^8.1.3",
    "@nativescript/datetimepicker": "^2.1.6",
    "@nativescript/geolocation": "^7.2.1",
    "@nativescript/iqkeyboardmanager": "^2.0.0",
    "@nativescript/theme": "^3.0.1",
    "@nstudio/nativescript-pulltorefresh": "^3.0.2",
    "jwt-decode": "^3.1.2",
    "luxon": "^1.27.0",
    "nativescript-appversion": "^1.4.4",
    "nativescript-fonticon": "^2.0.2",
    "nativescript-sqlite": "^2.8.4",
    "nativescript-sqlite-commercial": "file:../../../Downloads/nativescript-sqlite-commercial-master.tar.gz",
    "nativescript-sqlite-encrypted": "file:../../../Downloads/nativescript-sqlite-encrypted-master.tar.gz",
    "nativescript-ui-sidedrawer": "^10.0.1",
    "nativescript-vue": "^2.9.0",
    "nativescript-vue-lifecycle-hooks": "^1.0.0",
    "nativescript-vue-navigator": "^1.2.0",
    "semver": "^7.3.5",
    "vuelidate": "^0.7.6",
    "vuex": "^3.6.2"
  },
  "devDependencies": {
    "@nativescript/ios": "~8.1.0",
    "@nativescript/types": "~8.1.0",
    "@nativescript/webpack": "~5.0.0",
    "nativescript-vue-template-compiler": "~2.9.0",
    "sass": "^1.32.8",
    "util": "^0.12.4"
  },

Haven't tested the demo yet.

[Ed-Simmons]

Hi @NathanaelA,

I tried your Demo app, and sure enough, I didn't encounter the error there - very interesting (a.k.a confusing)!

I decided to try and create a simple demo app of my own this morning to try and isolate the issue, here's what I found:

• When launching the app for the first time, everything ran smoothly and the database was created as expected*
  • BUT the column created_at in my migrations table should have contained an empty string for each entry, instead they contain BLOB

Using:

INSERT INTO migrations (uuid, key, created_at) VALUES (?, ?, ?)

...with values like:

[ uuid(), key, '']

...resulted in:

Is this expected behaviour now? What is a BLOB??

...

• I can write data to the database, including null values, without any issues.
• I can also read that data from the database without issue, e.g:

SELECT * FROM users WHERE uuid = "123456"

...returns an array of records (of length 1) as expected:

[{
"uuid": "123456",
"email": "[email protected]",
"name": "ed",
"first_name": null,
"last_name": null,
"type": null
}]

The problem starts occurring when re-loading/ re-launching the app.

• On startup, I run through all of my migrations to check that they exist.
• The TypeError: Cannot read property 'toString' of null error occurs every time I run this check. This is the query string:

SELECT * FROM migrations WHERE key = "m2020_10_09_084300_create_users_table"

• As you can see from the first image, this check should return true, however I get the error instead.
• I've done lots of console logging, and can definitely confirm that the error is occurring in the Database.prototype.all() method, but I'm not sure where.

Re: iOS version: I've tried this on 14.4 (simulator) & 14.5 (simulator), both with the same result.

You're very welcome to this test database if it's helpful:
bug-test.sqlite.zip

You'll be able to see in the database that, as a result of the exists check failing every time, my migrator method is constantly adding new entries to the migrations table. Some with empty strings for created_at (as intended), but most with BLOB (unexpected).

Hope that's helpful! :)

[NathanaelA]

When launching the app for the first time, everything ran smoothly and the database was created as expected*
BUT the column created_at in my migrations table should have contained an empty string for each entry, instead they contain BLOB

Blob is a binary value (like a image). Interesting that it switched to blob, I did fix a iOS blob issue in the very latest sqlite that I discovered during my revamp/cleanup of the test framework but it was if you did query(sql, blob) instead of query(sql, [blob]) so it was a corner case that no one had run into, or they just used [ blob ] since all the documented examples shows params
being passed in as an array not as a single value.

Can you share a database with actual values in it, or tell me about how many records are in the migrations table, I can generate some dummy data if I have an idea.

[Ed-Simmons]

You're very welcome to this test database if it's helpful:
bug-test.sqlite.zip

The database I shared yesterday (towards the bottom of the comment) has lots of values on the migrations table, and one value in the users table - is that not usable?

[teha-at]

@Ed-Simmons thanks for helping me out with a sample.

I tested my app with iOS 15 and unfortunately get the same error.

[NathanaelA]

@Ed-Simmons - Sorry, for some reason I thought the table was empty. I do see some values in it. I'll take a look at it. 👍

On the blob issue:
Out of curiosity can you show me the query you used to add records to the migration... As I see the same thing that the created_at fields have "blob" values that are 5 or 6 binary characters long. Curious to know what you were trying to write to it, vs what got written to it.

[Ed-Simmons]

@NathanaelA I was trying to write an empty string into the created_at column:

CREATE TABLE IF NOT EXISTS migrations (
uuid TEXT(36) NOT NULL,
key TEXT(255) NOT NULL,
created_at TEXT(255) NOT NULL
);

INSERT INTO migrations (uuid, key, created_at) VALUES (?, ?, ?)

[ uuid, key, '' ]

Apologies for the delayed reply, I've been on vacation.

Just double checking (as I'm sure you're already aware), have you seen this excerpt from the NativeScript blog for the @8.1.0 update?

@nativescript/ios 8.1
In 8.1, several memory optimizations have been included, as well as an update of the internal v8 engine to 9.2.230.18.

In particular c string parameter handling has been optimized which will improve memory footprints of applications using SQLite databases.

For a full list of changes, see the full changelog here

Surely it's one of these changes that's causing the issue? SQLite databases are explicitly mentioned.

[timdoege]

I also get the same error on iOS runtime 8.1, and was just to post the same from the iOS 8.1 release notes ;)

[Ed-Simmons]

@NathanaelA I agree with @teha-at's initial observation, the error occurs in the _getNativeResult method, although I'm seeing the error in the text switch case.

Taking my migrations table as an example, when looping through the columns (uuid, key, and created_at) which should all be TEXT, the first two columns are read successfully, and the error occurs on the last one.

Indeed, as you correctly pointed out, created_at has been stored as a binary BLOB as opposed to an empty string:

CONSOLE LOG: [ _getNativeResult switch case 3 TEXT ] [ statement, column ] <Pointer: 0x7fb0502fbb80> 0
CONSOLE LOG: [ _getResults ] [ data ] {
"uuid": "d6878558-05cb-48df-87cb-3d692151c653"
}
CONSOLE LOG: [ _getNativeResult switch case 3 TEXT ] [ statement, column ] <Pointer: 0x7fb0502fbb80> 1
CONSOLE LOG: [ _getResults ] [ data ] {
"uuid": "d6878558-05cb-48df-87cb-3d692151c653",
"key": "m2020_10_07_102228_create_annoucements_table"
}
CONSOLE LOG: [ _getNativeResult switch case 3 TEXT ] [ statement, column ] <Pointer: 0x7fb0502fbb80> 2
CONSOLE LOG: (NATIVESCRIPT-ORM) [ERROR] TypeError: Cannot read property 'toString' of null
CONSOLE LOG: (NATIVESCRIPT-ORM) [ERROR] {SQL} SELECT * FROM migrations WHERE key = "m2020_10_07_102228_create_annoucements_table"

If I edit the database manually, replacing the binary blob with an empty string as intended, the error disappears:

...
CONSOLE LOG: [ _getNativeResult switch case 3 TEXT ] [ statement, column ] <Pointer: 0x7fea380bbb80> 2
CONSOLE LOG: [ _getResults ] [ data ] {
"uuid": "3f3ba8fa-c3f9-4014-ae45-98ee3cd833e6",
"key": "m2020_10_09_084300_create_users_table",
"created_at": ""
}

So, that leads me to think that something is going wrong when writing empty strings to the database. I shall keep looking...

[NathanaelA]

I'll update my test suite to send a empty string to a text field to see if I can duplicate it.

[NathanaelA]

Just a FYI: I am able to duplicate it. I may have a fix, I have released v2.8.6 which has a Patch that fixes it for me...

[Ed-Simmons]

Yay! Looks good to me, everything's working as expected now. Thanks so much!!

[timdoege]

Thanks, @NathanaelA - 2.8.6 fixes this for me, also!

[teha-at]

I updated to 2.8.6 and the error is still there. I'll try to get deep in my code/db to look if I have another character which make this problem.
@Ed-Simmons thanks a lot for your effort!

[sb-arzan]

Same issue here.

"TypeError: null is not an object (evaluating 'NSString.stringWithUTF8String(sqlite3_column_text(statement, column)).toString')" {}

Some Text fields are just stored As BLOB.

[NathanaelA]

@sb-arzan -

1. Are you using the latest version? v2.8.6 (or later)?
2. Can you give me a query that causes this?

[sb-arzan]

@NathanaelA

1. Are you using the latest version? v2.8.6 (or later)?
2. Can you give me a query that causes this?

https://drive.google.com/file/d/1f3wXsgQN6pDgivSTUSyOY1k6oMHg_iY2/view?usp=sharing

You can find the database on root directory called "demo2.sqlite"

The query is on ItemDetailComponent constructor.

ios simulator: Ipad Pro 9.7 (iOS 15)

[sb-arzan]

@NathanaelA

Is there any update regarding the BLOB issue ?

[NathanaelA]

Sorry, I have not had a change to investigate your project. I just looked at the package.json right now to verify you were using the latest (and you are. You should have let me know you purchased a commercial license, as that would have bumped the priority dramatically. I'll be taking a look at it sometime today.

[NathanaelA]

@sb-arzan - The fix that I released does NOT fix already corrupted data in your existing SQLite files caused by the NativeScript iOS NS8.1 engine changes. All I did was work around the data corruption issue, so that data corruption shouldn't happen any more since the NativeScript team hasn't fixed the underlying issue caused by NativeScript/ios#127.

The reason you are having issues is because your record with id#: 30084897 already has a corrupted "blob" record in the text value.

If you are worried about already deployed applications that might have corrupted data, I could potentially update the reading routine to filter out this corrupted data. We could add code to check for null (which shouldn't happen) and replace it with '' on reading.

[sb-arzan]

@NathanaelA thanks for your assistance.

We do have the commercial version. But as you also said, the corruption had happened in one of the older versions.

Again, thank you for your assistance.

[sb-arzan]

@NathanaelA The issue came back as we use empty strings in some columns.

Each time an empty string is inserted, we can see there is A BLOB instead and reading fails with Type null error.

We are using version 2.8.6 with the latest Nativescript version.

[NathanaelA]

@sb-arzan -

1. can you verify there isn't more than one copy of SQLite in your project, it is possible something else is bringing in an old version.
2. Can you manually go into the node_modules/nativescript-sqlite folder, and open up the package.json and verify it is using 2.8.6?
3. Can you give me a sample query that causes the issue, I have a test query that adds a empty string and my code seems to handle it fine.

[sb-arzan]

@NathanaelA

Thanks for the quick response.

Currently we are using our own workaround

• https://github.com/sablono/nativescript-sqlite

We wrapped all toString calls with try catch and return empty string. In other places, instead of writing empty string, we write NULL.

I'll try to provide a repo with a simple reproduce of issue.