I am a software and security engineer based in -Recife with 8+ +Recife with 10+ years of experience. Since 2020, I’m also building Unxpose a solution that simplifies cybersecurity for busy companies.
@@ -117,23 +118,10 @@From a53a877bfb74d8eaf5dbc21db7d052a065d52b37 Mon Sep 17 00:00:00 2001
From: Patrick Costa I am a software and security engineer based in
-Recife with 8+
+Recife with 10+
years of experience. Since 2020, I’m also building
Unxpose a solution that
simplifies cybersecurity for busy companies.References
rel="me"
href="https://infosec.exchange/@patrickrbc"
target="_blank"
- title="patrickrbc"
+ title=""
>
infosec.exchange/@patrickrbc
- References
rel="me"
href="https://infosec.exchange/@patrickrbc"
target="_blank"
- title="patrickrbc"
+ title=""
>
infosec.exchange/@patrickrbc
- References
rel="me"
href="https://infosec.exchange/@patrickrbc"
target="_blank"
- title="patrickrbc"
+ title=""
>
infosec.exchange/@patrickrbc
- Finding interesting stuff
rel="me"
href="https://infosec.exchange/@patrickrbc"
target="_blank"
- title="patrickrbc"
+ title=""
>
infosec.exchange/@patrickrbc
- The social media trap
infosec.exchange/@patrickrbc
- Conclusion
rel="me"
href="https://infosec.exchange/@patrickrbc"
target="_blank"
- title="patrickrbc"
+ title=""
>
infosec.exchange/@patrickrbc
- 404
rel="me"
href="https://infosec.exchange/@patrickrbc"
target="_blank"
- title="patrickrbc"
+ title=""
>
infosec.exchange/@patrickrbc
- About
Security advisories
rel="me"
href="https://infosec.exchange/@patrickrbc"
target="_blank"
- title="patrickrbc"
+ title=""
>
infosec.exchange/@patrickrbc
-
I’m not talking about UI testing here, so assume everything I’m saying is about server-side JS. If you’re happy with Jest, don’t switch! I’m not trying to convince anyone. I’m just sharing my experience because I bet there are people out there in the same situation as I was.
-<p>The decision to go with Jest wasn’t well thought out. At the time, I wasn’t heavily involved in software development, so I opted for the well-known, fully-fledged test framework. It seemed promising initially, with its great API.</p> +The decision to go with Jest wasn’t well thought out. At the time, I wasn’t heavily involved in software development, so I opted for the well-known, fully-fledged test framework. It seemed promising initially, with its great API.
-<p>However, after hundreds of tests, things started to break a lot. Memory leaks began to surface, the number of hacky flags multiplied, and visiting Jest’s issues tab became routine.</p> +However, after hundreds of tests, things started to break a lot. Memory leaks began to surface, the number of hacky flags multiplied, and visiting Jest’s issues tab became routine.
-<h2 id="the-problem">The problem</h2> +Jests loads modules over and over again, leading to a memory leak. It is designed in a way that makes those problems arise very likely, specially in backend projects that are getting big. I’m not going to try to describe the kind of issues here, but you can find them pretty easy there. I’m just going to leave some leads for the poor users that are still fighting this battle.
-<ol> - <li> - <p><strong>jest —logHeapUsage</strong> -<br /> -As I’ve mentioned before, <a href="https://github.com/jaredjj3/jest-memory-leak-demo">memory</a> <a href="https://github.com/smolijar/jest-is-a-rude-needy-clown-and-eats-lot-of-memory?tab=readme-ov-file">leaks</a> <a href="https://github.com/jestjs/jest/issues/12142">will</a> <a href="https://github.com/jestjs/jest/issues/7311">be</a> <a href="https://github.com/jestjs/jest/issues/7874">your</a> <a href="https://github.com/jestjs/jest/pull/8331">best</a> <a href="https://github.com/jestjs/jest/issues/11956">friend</a>, so you’d better watch your heap usage to spot sudden growth.</p> - </li> - <li> - <p><strong>jest —maxWorkers=50%</strong> -<br /> -Some <a href="https://ivantanev.com/make-jest-faster/">benchmarks</a> show that running with this configuration might make your tests run up to 20% faster. However, you have to test it in your project; some folks said it made things worse.</p> - </li> - <li> - <p><strong>jest —runInBand</strong> -<br /> -This command runs all tests serially in the current process instead of creating a worker pool of child processes. They say it’s useful for debugging, but oddly enough, some people reported that it can actually improve performance.</p> - </li> - <li> - <p><strong>jest —changedSince</strong> -<br /> -If you run a workflow for each PR update, your tests will run a LOT. So, if you have limited compute minutes or a constrained CI pipeline, this flag can significantly reduce the time your PR workflows take.</p> - </li> - <li> - <p><strong>jest-slow-test-reporter</strong> -<br /> -You can use this reporter to view which tests are the slowest in your project and start tackling them first.</p> - </li> - <li> - <p><strong>Exposing Node.js gargabe collector</strong> -<br /> -In some cases, running Node with the flag <strong><code class="language-plaintext highlighter-rouge">--expose-gc</code></strong> seems to handle memory leaks better.</p> - </li> -</ol> +jest —logHeapUsage
+
+As I’ve mentioned before, memory leaks will be your best friend, so you’d better watch your heap usage to spot sudden growth.
jest —maxWorkers=50%
+
+Some benchmarks show that running with this configuration might make your tests run up to 20% faster. However, you have to test it in your project; some folks said it made things worse.
jest —runInBand
+
+This command runs all tests serially in the current process instead of creating a worker pool of child processes. They say it’s useful for debugging, but oddly enough, some people reported that it can actually improve performance.
jest —changedSince
+
+If you run a workflow for each PR update, your tests will run a LOT. So, if you have limited compute minutes or a constrained CI pipeline, this flag can significantly reduce the time your PR workflows take.
jest-slow-test-reporter
+
+You can use this reporter to view which tests are the slowest in your project and start tackling them first.
Exposing Node.js gargabe collector
+
+In some cases, running Node with the flag --expose-gc
seems to handle memory leaks better.
Some of these strategies significantly reduced the running time during this period. However, the process of learning and implementing them came at the expense of delivery time, which is ultimately more critical.
-<p>The tests were so sluggish that I resorted to running them only on the module we were currently developing, then solely on the changed modules in the PR, and finally, all tests were run only when merging to the main branch. Unfortunately, this approach resulted in delays in identifying bugs.</p> +The tests were so sluggish that I resorted to running them only on the module we were currently developing, then solely on the changed modules in the PR, and finally, all tests were run only when merging to the main branch. Unfortunately, this approach resulted in delays in identifying bugs.
-<p>The tests were so time-consuming that I found myself hesitating to write tests for certain features, worrying about the additional build process time they would cause. At this point, I realized it was time to make the switch.</p> +The tests were so time-consuming that I found myself hesitating to write tests for certain features, worrying about the additional build process time they would cause. At this point, I realized it was time to make the switch.
-<h2 id="switching-to-mocha">Switching to Mocha</h2> +I used Mocha a decade ago and it was awesome. So, I thought transitioning back to it would be smooth sailing. Over the past years, I’ve been seeing people throwing out the idea of switching from Jest to Mocha, and always I found it funny. I remember there were a lot of guides and people talking about migrating from Mocha to Jest. Like me, most of people would assume the newer tool would have a better or at least similar performance.
-<p>The migration was much easier, than expected. A couple of replace cases and and less than an hour refactoring some code. The tricker part was the mock engine, which isn’t included in Mocha.</p> +The migration was much easier, than expected. A couple of replace cases and and less than an hour refactoring some code. The tricker part was the mock engine, which isn’t included in Mocha.
-<p>I could have used <a href="https://sinonjs.org/">Sinon.js</a> for that, but I really like the idea of one day not relying on any testing libraries. I even considered using only the new Node.js built-in test runner, but it’s not quite there yet for me. So, I decided to stick with just the built-in <a href="https://nodejs.org/api/test.html#class-mocktracker">MockTracker</a>.</p> +I could have used Sinon.js for that, but I really like the idea of one day not relying on any testing libraries. I even considered using only the new Node.js built-in test runner, but it’s not quite there yet for me. So, I decided to stick with just the built-in MockTracker.
-<p>Trying things out was mind-blowing. Single tests that were taking 3 seconds with Jest would run in less than 200ms with Mocha. It shouldn’t be a surprise – what I was running shouldn’t take so long, but I had gotten used to that slowness. In the end, our test running time reduced from more than 12 minutes to less than 40 seconds.</p> +Trying things out was mind-blowing. Single tests that were taking 3 seconds with Jest would run in less than 200ms with Mocha. It shouldn’t be a surprise – what I was running shouldn’t take so long, but I had gotten used to that slowness. In the end, our test running time reduced from more than 12 minutes to less than 40 seconds.
-<p>The speed of Mocha helped us uncover hidden bugs that would occasionally fail tests because they only occurred under very specific conditions – conditions that were unlikely to be met in Jest due to its slower performance.</p> +The speed of Mocha helped us uncover hidden bugs that would occasionally fail tests because they only occurred under very specific conditions – conditions that were unlikely to be met in Jest due to its slower performance.
-<h2 id="conclusion">Conclusion</h2> +I still use Jest on some smaller repositories that I maintain, and I’m not mad enough to migrate them until they become a problem. However, for future projects, I’ll definitely opt for Mocha or the Node.js test runner.
-<p>The thing is, even if there’s a way to optimize Jest and run thousands of tests in a reasonable time, there’s something wrong when simply switching the testing framework results in significantly faster performance. Do you agree? Do you have any similar experiences? I’d love to hear about them.</p>The thing is, even if there’s a way to optimize Jest and run thousands of tests in a reasonable time, there’s something wrong when simply switching the testing framework results in significantly faster performance. Do you agree? Do you have any similar experiences? I’d love to hear about them.
]]>You can indeed make money producing content online. It is also true that social media platforms can boost your reach. However, the majority of the population is just consuming and creating poor content that will never lead to a financial -reward.</p> +reward.
-<p>I would consider myself in the above situation. I’m not in a fair trade with +I would consider myself in the above situation. I’m not in a fair trade with social media. Not that I ever felt I could be. But the fear of missing out makes -me play the game without even thinking about it.</p> +me play the game without even thinking about it.
-<p>Most of the value generated from our actions is turned into revenue for the big +Most of the value generated from our actions is turned into revenue for the big techs. In exchange for that, you will be gifted with a subtle dependency on the -feedback you could receive if you saw something good enough.</p> +feedback you could receive if you saw something good enough.
-<p>If you can’t escape from it, maybe it’s a good idea to spend some time +If you can’t escape from it, maybe it’s a good idea to spend some time implementing filters to avoid the consumption of unhealthy content. You need to -feed yourself, just not with junky food.</p>
Enumerating subdomains with wildcard records is tricky but not impossible, here are some tips. Also, don’t trust wildcards as a security mechanism for hiding -sensitive apps.</p> +sensitive apps.
-<h1 id="the-problem">The problem</h1> +If you did some subdomain brute-force enumeration in the wild you already bumped into a record that resolves for any type of prefix. This is called a wildcard record and it can be configured by inserting a record entry with a -label “<strong>*</strong>”. This record will also resolve for other sublevels unless it is -inhibited by another record entry.</p> +label “*”. This record will also resolve for other sublevels unless it is +inhibited by another record entry.
-<p>Many companies use wildcard records as part of their architecture. A well-known +Many companies use wildcard records as part of their architecture. A well-known example is Slack which uses it for their workspaces. For example, today I asked my favorite DNS server to resolve the following records and got the same IP -address:</p> +address:
-<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>shopify.enterprise.slack.com 18.231.0.250 +shopify.enterprise.slack.com 18.231.0.250
enterprise.slack.com 18.231.0.250
big-name-non-existent.slack.com 18.231.0.250
-</code></pre></div></div>
+
In this case you might conclude that there is a wildcard record +*.slack.com and maybe we should ignore this domain in your subdomain enumeration. However, you could end up missing something like -<strong>status.slack.com</strong> which does not resolve to this address. Instead it has a -CNAME pointing to another infrastructure that could be interesting to you.</p> +status.slack.com which does not resolve to this address. Instead it has a +CNAME pointing to another infrastructure that could be interesting to you.
-<p>It is curious how often subdomain enumeration tools mess up or do not handle +It is curious how often subdomain enumeration tools mess up or do not handle this kind of behaviour. Many times the wildcard records are just dropped without any further check. The problem is that you might lose some interesting -apps by discarding them .</p> +apps by discarding them .
-<p>With that in mind, adding a wildcard record can be a tempting strategy to hide +With that in mind, adding a wildcard record can be a tempting strategy to hide your own services like a needle in the haystack. I can’t blame anyone for doing -that, but just keep in mind that this is not going to save you for long.</p> +that, but just keep in mind that this is not going to save you for long.
-<h1 id="finding-interesting-stuff">Finding interesting stuff</h1> +Thinking about how to make a better reconnaissance one could try to overcome this problem by treating enumeration in wildcard records differently. The response returned by the wildcard could be stored (sorted if it is multiple entries) and every subsequent DNS response would be compared with this one. -Everytime we find a new response it would be saved in a map structure.</p> +Everytime we find a new response it would be saved in a map structure.
-<p>This would make sure we have at least one subdomain that points to that new -location that we found. However, <em>the world ain’t all sunshine and rainbows</em> +This would make sure we have at least one subdomain that points to that new +location that we found. However, the world ain’t all sunshine and rainbows and we could obviously have a different application sitting on a machine that -will only show up when we set a specific Host header in the HTTP request.</p> +will only show up when we set a specific Host header in the HTTP request.
-<p>Therefore, this is just something you could use to have more places to look for +Therefore, this is just something you could use to have more places to look for security vulnerabilities. There are many other more edgy cases (for example when including CNAME) that can happen when trying to find assets using DNS. I -hope I can dig into that more in future posts.</p> +hope I can dig into that more in future posts.
-<p>Do you have any tips for finding apps on records with wildcard?</p>Do you have any tips for finding apps on records with wildcard?
]]>Firstly, make sure you have extracted the zip file. Using the file command we +can verify that the firmware was compiled for Linux MIPS.
-<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ file fw_NPLUG_1_0_0_14.bin +$ file fw_NPLUG_1_0_0_14.bin
fw_NPLUG_1_0_0_14.bin: u-boot legacy uImage, Linux Kernel Image, Linux/MIPS, OS
Kernel Image (lzma), 1731916 bytes, Wed Oct 12 17:28:28 2016, Load Address:
0x80000000, Entry Point: 0x802CB000, Header CRC: 0xEC8AD091, Data CRC:
0x3A3E5EAC
-</code></pre></div></div>
+
There is not much you can do with this file as it is. In order to extract useful stuff from it we need to find the offset of known file format signatures inside the binary. Once you know the offset you could extract the chunks using -the Linux <strong>dd</strong> tool.</p> +the Linux dd tool.
-<p>However, it is not necessary to do that manually, instead, we will use a tool -called <a href="https://github.com/ReFirmLabs/binwalk"><em>Binwalk</em></a>. This tool was created -by <a href="http://www.devttys0.com/">Craig Heffner</a> and it helps a lot in the process of extracting file systems -from firmware files.</p> +However, it is not necessary to do that manually, instead, we will use a tool +called Binwalk. This tool was created +by Craig Heffner and it helps a lot in the process of extracting file systems +from firmware files.
-<h1 id="using-binwalk">Using <em>Binwalk</em></h1> +Running Binwalk against the firmware without specifying any options would +give us the following output:
-<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>DECIMAL HEXADECIMAL DESCRIPTION +DECIMAL HEXADECIMAL DESCRIPTION
--------------------------------------------------------------------------------
-0 0x0 uImage header, header size: 64 bytes, header CRC: 0xEC8AD091, created: 2016-10-12 17:28:28, image size: 1731916 bytes, Data Address: 0x80000000, Entry Point: 0x802CB000, data CRC: 0x3A3E5EAC, OS: Linux, CPU: MIPS, image type: OS Kernel Image, compression type: lzma, image name: "Linux Kernel Image"
+0 0x0 uImage header, header size: 64 bytes, header CRC: 0xEC8AD091, created: 2016-10-12 17:28:28, image size: 1731916 bytes, Data Address: 0x80000000, Entry Point: 0x802CB000, data CRC: 0x3A3E5EAC, OS: Linux, CPU: MIPS, image type: OS Kernel Image, compression type: lzma, image name: "Linux Kernel Image"
64 0x40 LZMA compressed data, properties: 0x5D, dictionary size: 33554432 bytes, uncompressed size: 3829956 bytes
-</code></pre></div></div>
+
That means the binary has a header and some compressed data starting at offset 0x40. This assumption might be true or not, the tool is just trying to interpret the bytes in the file. -Using the <strong>xxd</strong> tool we can examine the first bytes from the binary which -refer to the header:</p> +Using the xxd tool we can examine the first bytes from the binary which +refer to the header:
-<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ xxd -l 64 fw_NPLUG_1_0_0_14.bin +$ xxd -l 64 fw_NPLUG_1_0_0_14.bin
00000000: 2705 1956 ec8a d091 57fe 72bc 001a 6d4c '..V....W.r...mL
-00000010: 8000 0000 802c b000 3a3e 5eac 0505 0203 …..,..:>^.....
+00000010: 8000 0000 802c b000 3a3e 5eac 0505 0203 …..,..:>^.....
00000020: 4c69 6e75 7820 4b65 726e 656c 2049 6d61 Linux Kernel Ima
00000030: 6765 0000 0000 0000 0000 0000 0000 0000 ge..............
-</code></pre></div></div>
+
Binwalk assumed the following chunk was related to some LZMA compressed data
+because of the signature 5d 00 00 00 02
. This signature might
+vary depending
+on the compression level used, but apparently it always start with 5d 00 00
.
+Here is the start of the compressed data:
[...]
00000040: 5d00 0000 02c4 703a 0000 0000 0000 006f ].....p:.......o
-00000050: fdff ffa3 b7ff 473e 4815 7239 6151 b892 …...G>H.r9aQ..
-00000060: 28e6 a386 07f9 eee4 1e82 d32f c53a 3c01 (........../.:<.
+00000050: fdff ffa3 b7ff 473e 4815 7239 6151 b892 …...G>H.r9aQ..
+00000060: 28e6 a386 07f9 eee4 1e82 d32f c53a 3c01 (........../.:<.
00000070: 4bb1 7ec9 8a8a 4d2f a30d d97f a6e3 8c23 K.~...M/.......#
00000080: 1153 e059 18c5 758a e277 f886 f31f 9269 .S.Y..u..w.....i
00000090: 516e f9a2 84d4 b228 94f8 6af5 1bf9 141e Qn.....(..j.....
[...]
-</code></pre></div></div>
+
The -e (--extract)
flag, will extract the files identified during a
+signature scan and the -M (--matryoshka)
flag will perform this
+extraction recursively. There is also another really useful option in Binwalk
which allows an entropy analysis of the file. This can be helpful when
-<em>Binwalk</em>
-doesn’t work properly on the target file and you have to do a manual analysis.</p>
+Binwalk
+doesn’t work properly on the target file and you have to do a manual analysis.
For example, taking a step back, if we used Binwalk extraction only once, we
+would find a file named 40 (this number refers to its offset in hexadecimal).
+Using Binwalk with the -E (--entropy)
flag in this file will pop the
+following line chart (generated with pyqtgraph).
+
The Y axis is the entropy and the X axis refers to the offset in that file. With this graph we can infer that there are some readable stuff in that file but there are some other regions with high entropy. A high entropy in a file -probably means compressed or encrypted data, but it could also be just garbage.</p> +probably means compressed or encrypted data, but it could also be just garbage.
-<p>The signature scan for this file would give us the following output:</p> +The signature scan for this file would give us the following output:
-<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ binwalk 40 +$ binwalk 40
DECIMAL HEXADECIMAL DESCRIPTION
--------------------------------------------------------------------------------
-2658360 0x289038 Linux kernel version "2.6.21 (root@linux-qxix)
-(gcc version 3.4.2) #1086 Thu Oct 13 01:28:23 CST 2016"
+2658360 0x289038 Linux kernel version "2.6.21 (root@linux-qxix)
+(gcc version 3.4.2) #1086 Thu Oct 13 01:28:23 CST 2016"
2659360 0x289420 CRC32 polynomial table, little endian
2686144 0x28FCC0 SHA256 hash constants, little endian
2728378 0x29A1BA Unix path: /mru/rcvseq/sendseq/lns debug
reorderto
2736880 0x29C2F0 Unix path: /etc/Wireless/RT2860AP/RT2860AP.dat
-2737968 0x29C730 XML document, version: "1.0"
+2737968 0x29C730 XML document, version: "1.0"
2870768 0x2BCDF0 CRC32 polynomial table, little endian
3059712 0x2EB000 LZMA compressed data, properties: 0x5D,
dictionary size: 1048576 bytes, uncompressed size: 2943488 bytes
-</code></pre></div></div>
+
Performing another extraction with Binwalk in the file 40 will output +another file named 2EB000. We can generate the entropy analysis line chart +for the new file too.
-<p><img src="/assets/nplug/2EB000.png" alt="Entropy of file 2EB000" /> -<br /></p> +
+
This time we can see a lot of regions in the file with data that is probably readable. Not surprisingly, we have found the file system of the firmware. Of course, we could have simply extracted everything with just one command like -the following demonstration:</p> +the following demonstration:
-<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ binwalk -eM fw_NPLUG_1_0_0_14.bin +$ binwalk -eM fw_NPLUG_1_0_0_14.bin
$ cd fw_NPLUG_1_0_0_14.bin.extracted/_40.extracted/_2EB000.extracted/cpio-root/
$ tree
.
├── bin
-│ ├── ash -> busybox
+│ ├── ash -> busybox
│ ├── ate
│ ├── ated
│ ├── busybox
-│ ├── cat -> busybox
-│ ├── chmod -> busybox
-│ ├── cp -> busybox
-│ ├── date -> busybox
+│ ├── cat -> busybox
+│ ├── chmod -> busybox
+│ ├── cp -> busybox
+│ ├── date -> busybox
│ ├── default.cfg
│ ├── dnrd
-│ ├── echo -> busybox
-│ ├── hostname -> busybox
+│ ├── echo -> busybox
+│ ├── hostname -> busybox
│ ├── httpd
│ ├── inadyn
│ ├── iptables
│ ├── iwconfig
│ ├── iwpriv
-│ ├── kill -> busybox
-│ ├── ls -> busybox
+│ ├── kill -> busybox
+│ ├── ls -> busybox
│ ├── miniupnpd
-│ ├── mkdir -> busybox
-│ ├── mount -> busybox
+│ ├── mkdir -> busybox
+│ ├── mount -> busybox
│ ├── msg
│ ├── mtd_write
│ ├── netctrl
-│ ├── nvram_clear -> ralink_init
-│ ├── nvram_commit -> ralink_init
-│ ├── nvram_get -> ralink_init
-│ ├── nvram_set -> ralink_init
-│ ├── nvram_show -> ralink_init
-│ ├── ping -> busybox
+│ ├── nvram_clear -> ralink_init
+│ ├── nvram_commit -> ralink_init
+│ ├── nvram_get -> ralink_init
+│ ├── nvram_set -> ralink_init
+│ ├── nvram_show -> ralink_init
+│ ├── ping -> busybox
│ ├── pppd
-│ ├── ps -> busybox
+│ ├── ps -> busybox
│ ├── ralink_init
-│ ├── rm -> busybox
-│ ├── sh -> busybox
+│ ├── rm -> busybox
+│ ├── sh -> busybox
│ ├── sntp
│ ├── switch
│ ├── tenda_chck_conn
@@ -427,34 +427,34 @@ $ tree
│ │ └── RT5350_AP_1T1R_V1_0.bin
│ └── xml
├── home
-├── init -> bin/busybox
+├── init -> bin/busybox
├── lib
│ ├── ipsec
│ ├── ld-uClibc-0.9.28.so
-│ ├── ld-uClibc.so.0 -> ld-uClibc-0.9.28.so
+│ ├── ld-uClibc.so.0 -> ld-uClibc-0.9.28.so
│ ├── libcrypt-0.9.28.so
-│ ├── libcrypt.so -> libcrypt-0.9.28.so
-│ ├── libcrypt.so.0 -> libcrypt-0.9.28.so
-│ ├── libc.so -> libuClibc-0.9.28.so
-│ ├── libc.so.0 -> libuClibc-0.9.28.so
+│ ├── libcrypt.so -> libcrypt-0.9.28.so
+│ ├── libcrypt.so.0 -> libcrypt-0.9.28.so
+│ ├── libc.so -> libuClibc-0.9.28.so
+│ ├── libc.so.0 -> libuClibc-0.9.28.so
│ ├── libdl-0.9.28.so
-│ ├── libdl.so -> libdl-0.9.28.so
-│ ├── libdl.so.0 -> libdl-0.9.28.so
+│ ├── libdl.so -> libdl-0.9.28.so
+│ ├── libdl.so.0 -> libdl-0.9.28.so
│ ├── libiw.so.29
│ ├── libnvram-0.9.28.so
-│ ├── libnvram.so -> libnvram-0.9.28.so
-│ ├── libnvram.so.0 -> libnvram-0.9.28.so
+│ ├── libnvram.so -> libnvram-0.9.28.so
+│ ├── libnvram.so.0 -> libnvram-0.9.28.so
│ ├── libpthread-0.9.28.so
-│ ├── libpthread.so -> libpthread-0.9.28.so
-│ ├── libpthread.so.0 -> libpthread-0.9.28.so
+│ ├── libpthread.so -> libpthread-0.9.28.so
+│ ├── libpthread.so.0 -> libpthread-0.9.28.so
│ ├── libresolv-0.9.28.so
-│ ├── libresolv.so -> libresolv-0.9.28.so
-│ ├── libresolv.so.0 -> libresolv-0.9.28.so
+│ ├── libresolv.so -> libresolv-0.9.28.so
+│ ├── libresolv.so.0 -> libresolv-0.9.28.so
│ ├── libtenda_common.so
│ ├── libuClibc-0.9.28.so
│ ├── libutil-0.9.28.so
-│ ├── libutil.so -> libutil-0.9.28.so
-│ ├── libutil.so.0 -> libutil-0.9.28.so
+│ ├── libutil.so -> libutil-0.9.28.so
+│ ├── libutil.so.0 -> libutil-0.9.28.so
│ └── modules
│ ├── 2.6.21
│ │ └── kernel
@@ -467,546 +467,546 @@ $ tree
├── mnt
├── proc
├── sbin
-│ ├── arp -> ../bin/busybox
-│ ├── halt -> ../bin/busybox
-│ ├── ifconfig -> ../bin/busybox
-│ ├── init -> ../bin/busybox
-│ ├── insmod -> ../bin/busybox
-│ ├── lsmod -> ../bin/busybox
-│ ├── poweroff -> ../bin/busybox
+│ ├── arp -> ../bin/busybox
+│ ├── halt -> ../bin/busybox
+│ ├── ifconfig -> ../bin/busybox
+│ ├── init -> ../bin/busybox
+│ ├── insmod -> ../bin/busybox
+│ ├── lsmod -> ../bin/busybox
+│ ├── poweroff -> ../bin/busybox
│ ├── pppoe.sh
-│ ├── reboot -> ../bin/busybox
-│ ├── rmmod -> ../bin/busybox
-│ ├── route -> ../bin/busybox
-│ └── vconfig -> ../bin/busybox
+│ ├── reboot -> ../bin/busybox
+│ ├── rmmod -> ../bin/busybox
+│ ├── route -> ../bin/busybox
+│ └── vconfig -> ../bin/busybox
├── sys
├── tmp
├── usr
│ ├── bin
-│ │ ├── expr -> ../../bin/busybox
-│ │ ├── killall -> ../../bin/busybox
-│ │ └── top -> ../../bin/busybox
+│ │ ├── expr -> ../../bin/busybox
+│ │ ├── killall -> ../../bin/busybox
+│ │ └── top -> ../../bin/busybox
│ ├── codepages
│ └── sbin
-│ ├── brctl -> ../../bin/busybox
-│ └── telnetd -> ../../bin/busybox
+│ ├── brctl -> ../../bin/busybox
+│ └── telnetd -> ../../bin/busybox
└── var
41 directories, 185 files
-</code></pre></div></div>
-<h1 id="conclusion">Conclusion</h1>
+
In this post we learned how to extract the file system from a firmware file. Sometimes this task can be much harder than what was demonstrated here. A firmware file can be constructed in many different ways which makes it -difficult to analyze a bunch of them.</p> - -<p>In the next part I am planning to go through the file system trying to find -interesting stuff, stay tuned!</p> - -<h1 id="references">References</h1> -<ul> - <li><a href="https://github.com/ReFirmLabs/binwalk">Binwalk</a></li> - <li><a href="https://github.com/cscott/lzma-purejs/blob/master/FORMAT.md#lzma-file-format">Notes on the LZMA format</a></li> - <li><a href="http://www.devttys0.com/">/dev/ttys0</a></li> -</ul>
In the next part I am planning to go through the file system trying to find +interesting stuff, stay tuned!
+ +SPOILER ALERT: This is post will give the solutions for the first set of exercises, if you are planning to do it, stop reading this now and try them by yourself. However, if you are still unsure if this is for you or not, maybe try -skim reading through this post to see if you get interested.</p> +skim reading through this post to see if you get interested.
-<p>While resolving the first set I had mixed feelings of love and hatred. +While resolving the first set I had mixed feelings of love and hatred. Sometimes I was able to get the idea but ended up making a programming mistake that would keep me stuck for while in one challenge or another. They stated in their website that this is a “relatively easy” set of challenges with the except of one. I can’t tell you about the further sets but one challenge in -this set took me 2 days to get it right.</p> +this set took me 2 days to get it right.
-<p>I decided to write my solutions in Node.js and this was a good way to learn +I decided to write my solutions in Node.js and this was a good way to learn about some quirks of the language that I was not used to face in a common daily programming session. You might also want to choose a new programming language -to kill two birds with one stone.</p> +to kill two birds with one stone.
-<h1 id="challenge-1-convert-hex-to-base64">Challenge 1: Convert hex to base64</h1> +There is no mystery in here. If you were dealing with client-side JavaScript you would be looking for atob and btoa, but in Node.js you will want to be familiar with the Buffer API. Needless to -say, you shouldn’t include any third-party libraries to solve the challenges.</p> +say, you shouldn’t include any third-party libraries to solve the challenges.
-<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="kd">var</span> <span class="nx">result</span> <span class="o">=</span> <span class="nx">Buffer</span> - <span class="p">.</span><span class="k">from</span><span class="p">(</span><span class="dl">'</span><span class="s1">49276d206b696c6c696e6720796f757220627261696e206c696b65206120706f69736f6e6f7573206d757368726f6f6d</span><span class="dl">'</span><span class="p">,</span><span class="dl">'</span><span class="s1">hex</span><span class="dl">'</span><span class="p">)</span> - <span class="p">.</span><span class="nx">toString</span><span class="p">(</span><span class="dl">'</span><span class="s1">base64</span><span class="dl">'</span><span class="p">)</span></code></pre></figure> + -<p><br /></p> +This challenge will introduce you to XOR operation for cryptographic purposes. +The simple XOR cipher is a type of additive cipher that was used in the past to encrypt text, but it is clearly not safe since the ciphertext could be decrypted using character frequency analysis. On the other hand, the XOR operation is still frequently used in more complex ciphers, due to its -simplicity and performance.</p> +simplicity and performance.
-<p>The challenge asks for a function that takes two equal-length buffers and +The challenge asks for a function that takes two equal-length buffers and produces their XOR combination. A solution for this exercise would be to loop through one of the buffers doing a XOR with each correspondent byte from the -other buffer. The code that can do this will be shown further in this post.</p> +other buffer. The code that can do this will be shown further in this post.
-<p><br /></p> +Now that we already know how to cipher and decipher messages using XOR, things are going to become interesting. The challenge presents a hex encoded string that was XOR’d against a single character and require us to discover this key and decrypt the message. This can be achieved by brute-forcing the key and analysing the character frequency of the output messages. The output with the -best score is probably the message decrypted.</p> +best score is probably the message decrypted.
-<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="cm">/** + -<p>To calculate the score of a string according to the frequency of english +To calculate the score of a string according to the frequency of english letters you can find the percentual values on the Internet and sum them. Don’t forget to add the space character to your list, it took me some time to notice -that.</p> +that.
-<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="cm">/** + -<p>If everything works fine, we will get the decrypted message:</p> +If everything works fine, we will get the decrypted message:
-<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Cooking MC\'s like a pound of bacon -</code></pre></div></div> +Cooking MC\'s like a pound of bacon
+
This time we were given a file containing a lot of encrypted strings. The goal was to find the string that was encrypted with single-character XOR, but we know from the previous exercise that decrypting it would be really easy too. We can use the function from the previous challenge to brute-force each string as -a single-byte XOR cipher and find out the one with the best score.</p> +a single-byte XOR cipher and find out the one with the best score.
-<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="kd">var</span> <span class="nx">decrypted</span> <span class="o">=</span> <span class="p">[],</span> <span class="nx">result</span> + -<p>If everything works fine, we will get the decrypted message:</p> +If everything works fine, we will get the decrypted message:
-<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Now that the party is jumping\n -</code></pre></div></div> +Now that the party is jumping\n
+
This challenge is similar to the Challenge #2, but this time the message length +is bigger than the key length, so we will use the Vigenère +cipher. In this case, the key needs to be repeated to cipher the whole message. Maybe this challenge could be -positioned after the second one.</p> +positioned after the second one.
-<p>In order to reuse code, some of the functions were created in a separated file. +In order to reuse code, some of the functions were created in a separated file. Since XOR is a commonly used operation, I decided to create a function that will XOR two buffers repeating the smallest one which would be the key. This -function could be used with fixed or repeating-key XOR.</p> +function could be used with fixed or repeating-key XOR.
-<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="cm">/** + -<p>Ciphering the message with the above function and converting the result to a -hex string will give the expected output.</p> +Ciphering the message with the above function and converting the result to a +hex string will give the expected output.
-<p><br /></p> +“It is officially on, now.” This one took me a while to make it work right. They give you a file containing a message that is encrypted with repeating-xor and encoded in base64. This challenge is when you put together everything you learned until now. -Fortunately, they give you the steps to do it.</p> +Fortunately, they give you the steps to do it.
-<p>In order to get this one right, you will need to use something called the -<a href="https://en.wikipedia.org/wiki/Hamming_distance">Hamming distance</a>, which is +In order to get this one right, you will need to use something called the +Hamming distance, which is the number of different position between two symbols. In this case, we need to -calculate the number of differing bits between the two strings.</p> +calculate the number of differing bits between the two strings.
-<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="cm">/** + + +The first thing you need to do is to figure out the size of the key. Unless you have a crystal ball at home, you can start guessing. They suggest values from 2 -to 40.</p> +to 40.
-<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="cm">/** + -<p>If you are wondering what kind of test is that, no worries, they explain you +If you are wondering what kind of test is that, no worries, they explain you how to do it. It basically consists of getting the first two chunks of KEYSIZE bytes from the ciphertext, computing the Hamming distance between them, repeating this process a couple of times and normalizing the result. The KEYSIZE which gives the least distance between the chunks of bytes in the -ciphertext is probably the right one.</p> +ciphertext is probably the right one.
-<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="cm">/** + + +Once you have the KEYSIZE, you can use the following strategy to solve the +challenge:
+ +Using this key against the file will give you the lyrics of this +song.
+ +After solving the previous challenge the last two seemed like a walk in the park. This is an introduction to the block cipher -<a href="https://en.wikipedia.org/wiki/Advanced_Encryption_Standard">AES</a>. A <a href="https://en.wikipedia.org/wiki/Block_cipher#Rijndael_">block -cipher</a> is a +AES. A block +cipher is a deterministic algorithm which operates on fixed-length groups of bits. Examples of block ciphers are DES, RC5, Blowfish, AES. The following code will do the -job of decrypting a base64 encoded ciphertext in Node.js:</p> +job of decrypting a base64 encoded ciphertext in Node.js:
-<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="cm">/** + -<p>If everything works fine, we will get the same lyrics of the previous -challenge.</p> +If everything works fine, we will get the same lyrics of the previous +challenge.
-<p><br /></p> +This challenges requires you to find which one of the ciphertexts in a file was encrytpted with AES in ECB mode. The ECB mode is simplest mode of encryption in which the message is divided into blocks and they are independently encrypted. The problem with this mode is that a block is always encrypted into the same ciphertext block. This is exactly the weakness you need to abuse in order to detect which ciphertext was encrypted in ECB mode, you will count the -repetitions.</p> +repetitions.
-<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="nx">ciphertexts</span><span class="p">.</span><span class="nx">forEach</span><span class="p">((</span><span class="nx">ciphertext</span><span class="p">,</span> <span class="nx">index</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span> - <span class="kd">var</span> <span class="nx">cipherTextBlocks</span> <span class="o">=</span> <span class="nx">Util</span><span class="p">.</span><span class="nx">divideInBlocks</span><span class="p">(</span><span class="nx">ciphertext</span><span class="p">,</span> <span class="mi">16</span><span class="p">)</span> + -<p><br /></p> +The complete source code of the solutions you can find on my +Github. If you solved this first +set of exercises you have the knowledge of performing the following tasks:
-<ul> - <li>Convert between data base64, hex, binary</li> - <li>Detect and decrypt single-character XOR ciphers</li> - <li>Break repeating-key XOR ciphers</li> - <li>Detect and decipher AES in ECB mode</li> -</ul> +I am planning on doing the other sets soon. I will probably release the +solutions on Github and make my comments about them here. Stay tuned!
-<p><br /></p> +For the first part, we are going to understand the basics and learn how to collect information about our target. If you have any questions, please leave -a comment below so I can improve this article.</p> +a comment below so I can improve this article.
-<h1 id="what-is-a-firmware">What is a firmware?</h1> +The term “firmware” was firstly used in a magazine, published in 1967, and it was referring to the CPU microcode existent between the hardware and the software. Today, the term is more broadly used to refer to the BIOS, -bootloaders or embedded management systems.</p> +bootloaders or embedded management systems.
-<p>There are many different kinds of firmwares files. It could be a complete +There are many different kinds of firmwares files. It could be a complete operation system with custom drivers and user applications, or it could contain only the user applications. Sometimes, it could be composed only of partial updates for an application, for example, the front-end of an embedded web -server.</p> +server.
-<h1 id="why-should-we-do-this">Why should we do this?</h1> +SOHO routers are the front door of your home network, therefore, a security issue in this device might compromise all the connected clients. On the other hand, SOHO routers are known for having security issues that are widely exploited by malwares. In addition, the source code of the firmware is usually not public. Unless someone does a reverse engineering on it, we won’t be able -to know what is actually running inside it.</p> +to know what is actually running inside it.
-<p>Another motivation is learning. Reversing SOHO routers is a good way to study +Another motivation is learning. Reversing SOHO routers is a good way to study about common low-level vulnerabilities and reversing due to its simplicity -compared to modern systems.</p> +compared to modern systems.
-<h1 id="information-gathering">Information gathering</h1> +For our case study we are going to use an old wireless repeater named +NPLUG, previously offered by the Brazilian company Intelbras. Unfortunately, I couldn’t find this product on their website anymore, but this -is actually good for everyone. :)</p> +is actually good for everyone. :)
-<p>Firstly, it is good to gather as much information about the device as you can. +Firstly, it is good to gather as much information about the device as you can. This allows us to get a general idea of how it works and uncover some weakness without the need to look inside. An easy way to do that is looking for the FCC ID number of the device. The FCC ID is the registration number of that device in Federal Communications Commission of United States. Using this number you can -find more information like user manuals, specifications and internal pictures.</p> +find more information like user manuals, specifications and internal pictures.
-<p>Once you have that number you can search for it using sites like <a href="https://fccid.io/">FCC -ID.io</a> or the official <a href="https://www.fcc.gov/oet/ea/fccid">FCC</a> page. Depending on where are +Once you have that number you can search for it using sites like FCC +ID.io or the official FCC page. Depending on where are your device distributed you might not find a FCC ID number on it, but another -certification number instead.</p> +certification number instead.
-<p>In Brazil, Anatel is the agency responsible for registering and regulating electronic devices. +In Brazil, Anatel is the agency responsible for registering and regulating electronic devices. Taking a look at the back of our device, between other information, we can find a number near the Anatel logo. This is the Anatel’s certification number and, -fortunately, <a href="https://fccid.io/">FCC ID.io</a> has a database for that too.</p> +fortunately, FCC ID.io has a database for that too.
-<p><img src="/assets/nplug/nplug.png" alt="NPLUG wireless repeater" /></p> + -<p>I couldn’t find much information (beyond what was on the vendor’s website) +I couldn’t find much information (beyond what was on the vendor’s website) about the device using this number. However, if you are into embedded devices you might have noticed similarities between the NPLUG and other wireless repeaters. I had the same feeling and while looking at similar devices I -discovered a “cousin” of NPLUG, which is the <a href="https://www.aliexpress.com/item/English-Firmware-Tenda-A5S-Mini-Router-Pocket-WiFi-Wireless-Router-Client-Universal-Repeater-WISP-150Mbps-Ethernet/32751330210.html">Tenda A5s</a>.</p> +discovered a “cousin” of NPLUG, which is the Tenda A5s.
-<p>Searching for “<em>fcc id tenda a5s</em>”, I was able to find the <a href="https://fccid.io/V7TA5S">FCC -ID.io</a> page for the Tenda product and within it I could internal +Searching for “fcc id tenda a5s”, I was able to find the FCC +ID.io page for the Tenda product and within it I could internal pictures of the device. Looking at the information found I realized that both devices are almost the same. They were just sold with a different name, by a different company, in a different country. This is something that was previously pointed out by other security researchers. The code reuse between router manufacturers has propagated not only the same features but also the -same security holes between devices.</p> +same security holes between devices.
-<p><img src="/assets/nplug/internal.png" alt="Internal photos Tenda A5S" /></p> -<center> - <small> + +The vendor website usually provides a download section where you can get manuals and the firmware file. If this is not the case you might get lucky searching for it in your search engine. For this series we are going to use the firmware of the NPLUG wireless repeater -which can be acquired <a href="http://en.intelbras.com.br/sites/default/files/downloads/fw_nplug_1_0_0_14.zip">here</a>.</p> +which can be acquired here.
-<p>When none of the above options is available you need to extract the firmware image from the hardware using one of the techniques that will be briefly described on the following posts.</p> +When none of the above options is available you need to extract the firmware image from the hardware using one of the techniques that will be briefly described on the following posts.
-<h1 id="conclusion">Conclusion</h1> +This is all for now. I hope I had increased your appetite for what is coming next. Make sure you subscribe to the RSS feed and follow me on Twitter so you -can see when the Part 2 is released.</p> +can see when the Part 2 is released.
-<h1 id="references">References</h1> +