v1.0.0
projectdiscovery/katanav1.0.0Mar 20, 2023by ehsandeep
AI Summary
The major v1.0.0 release introduces a significant JSON output restructuring to include full request/response details, technology detection, and various new CLI flags for filtering and resolvers.
Key Highlights
- Breaking change: JSON output structure refactored to include request/response objects with full details
- Technology detection using Wappalyzer in JSON output
- Match and filter support for discovered endpoints
- Custom resolver input support
- Self-update option
Breaking Changes
- JSON output structure refactored to include request and response objects with full details (breaking change)
New Features
- Structured JSON output including request method, headers, body, status code, and technologies
- Technology detection via Wappalyzer
- Regex match and filter support (-mr, -fr)
- Custom resolver input support (-r)
- Self-update option (-up)
- Authenticated crawling flags (-chrome-data-dir)
Full Release Notes
## What's Changed
> **Warning**: breaking changes in json output structure.
<details>
<summary>previous format</summary>
```json
{
"timestamp": "2022-11-05T22:33:27.745815+05:30",
"endpoint": "https://www.iana.org/domains/example",
"source": "https://example.com",
"tag": "a",
"attribute": "href"
}
```
</details>
<details>
<summary>new format</summary>
```json
{
"timestamp": "2023-03-20T16:23:58.027559+05:30",
"request": {
"method": "GET",
"endpoint": "https://example.com",
"tag": "a",
"attribute": "href",
"source": "http://www.iana.org/domains/reserved",
"raw": "GET / HTTP/1.1\r\nHost: example.com\r\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36\r\nAccept-Encoding: gzip\r\n\r\n"
},
"response": {
"status_code": 200,
"headers": {
"accept_ranges": "bytes",
"expires": "Mon, 27 Mar 2023 10:53:58 GMT",
"last_modified": "Thu, 17 Oct 2019 07:18:26 GMT",
"content_type": "text/html; charset=UTF-8",
"server": "ECS (dcb/7EA3)",
"vary": "Accept-Encoding",
"etag": "\"3147526947\"",
"cache_control": "max-age=604800",
"x_cache": "HIT",
"date": "Mon, 20 Mar 2023 10:53:58 GMT",
"age": "331239"
},
"body": "<!doctype html>\n<html>\n<head>\n <title>Example Domain</title>\n\n <meta charset=\"utf-8\" />\n <meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\" />\n <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\" />\n <style type=\"text/css\">\n body {\n background-color: #f0f0f2;\n margin: 0;\n padding: 0;\n font-family: -apple-system, system-ui, BlinkMacSystemFont, \"Segoe UI\", \"Open Sans\", \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n \n }\n div {\n width: 600px;\n margin: 5em auto;\n padding: 2em;\n background-color: #fdfdff;\n border-radius: 0.5em;\n box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);\n }\n a:link, a:visited {\n color: #38488f;\n text-decoration: none;\n }\n @media (max-width: 700px) {\n div {\n margin: 0 auto;\n width: auto;\n }\n }\n </style> \n</head>\n\n<body>\n<div>\n <h1>Example Domain</h1>\n <p>This domain is for use in illustrative examples in documents. You may use this\n domain in literature without prior coordination or asking for permission.</p>\n <p><a href=\"https://www.iana.org/domains/example\">More information...</a></p>\n</div>\n</body>\n</html>\n",
"technologies": [
"Azure",
"Amazon ECS",
"Amazon Web Services",
"Docker",
"Azure CDN"
],
"raw": "HTTP/1.1 200 OK\r\nContent-Length: 1256\r\nAccept-Ranges: bytes\r\nAge: 331239\r\nCache-Control: max-age=604800\r\nContent-Type: text/html; charset=UTF-8\r\nDate: Mon, 20 Mar 2023 10:53:58 GMT\r\nEtag: \"3147526947\"\r\nExpires: Mon, 27 Mar 2023 10:53:58 GMT\r\nLast-Modified: Thu, 17 Oct 2019 07:18:26 GMT\r\nServer: ECS (dcb/7EA3)\r\nVary: Accept-Encoding\r\nX-Cache: HIT\r\n\r\n<!doctype html>\n<html>\n<head>\n <title>Example Domain</title>\n\n <meta charset=\"utf-8\" />\n <meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\" />\n <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\" />\n <style type=\"text/css\">\n body {\n background-color: #f0f0f2;\n margin: 0;\n padding: 0;\n font-family: -apple-system, system-ui, BlinkMacSystemFont, \"Segoe UI\", \"Open Sans\", \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n \n }\n div {\n width: 600px;\n margin: 5em auto;\n padding: 2em;\n background-color: #fdfdff;\n border-radius: 0.5em;\n box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);\n }\n a:link, a:visited {\n color: #38488f;\n text-decoration: none;\n }\n @media (max-width: 700px) {\n div {\n margin: 0 auto;\n width: auto;\n }\n }\n </style> \n</head>\n\n<body>\n<div>\n <h1>Example Domain</h1>\n <p>This domain is for use in illustrative examples in documents. You may use this\n domain in literature without prior coordination or asking for permission.</p>\n <p><a href=\"https://www.iana.org/domains/example\">More information...</a></p>\n</div>\n</body>\n</html>\n"
}
}
```
</details>
* Refactored katana core by @Mzack9999 in https://github.com/projectdiscovery/katana/pull/312
- for synchronous correlation between http request/response
- **added request/response data into json output**
- **added status code information in json output**
- updated json output structure
* Added **technology detection of crawled endpoints in json output using wappalyzer** by @Mzack9999 in https://github.com/projectdiscovery/katana/pull/294
```json
...
"technologies": [
"Azure",
"Amazon ECS",
"Amazon Web Services",
"Docker",
"Azure CDN"
],
...
```
* Added **match and filter support** for discovered endpoint by @ShubhamRasal in https://github.com/projectdiscovery/katana/pull/346
```console
-mr, -match-regex string[] regex or list of regex to match on output url (cli, file)
-fr, -filter-regex string[] regex or list of regex to filter on output url (cli, file)
```
* Added custom resolver input support by @ShubhamRasal in https://github.com/projectdiscovery/katana/pull/340
```console
-r, -resolvers string[] list of custom resolver (file or comma separated)
```
* Added self update option by @RamanaReddy0M in https://github.com/projectdiscovery/katana/pull/356
```console
-up, -update update katana to latest version
```
* Added support for `ufile` field based filter by @Mzack9999 in https://github.com/projectdiscovery/katana/pull/357
```console
katana -f ufile -u https://www.tesla.com
```
* Added `-chrome-data-dir` / `-cdd` flag to implement authenticated crawling #230 by @parthmalhotra in https://github.com/projectdiscovery/katana/pull/231
* Added examples using `rdn`, `fqdn`, etc. scope filter in readme by @olearycrew in https://github.com/projectdiscovery/katana/pull/288
* Fixed tls version error when using burp proxy by @Mzack9999 in https://github.com/projectdiscovery/katana/pull/311
* Fixed **issues with katana consuming available cpu** by @ShubhamRasal in https://github.com/projectdiscovery/katana/pull/362
* Fixed `Dockerfile` to avoid version mismatch by @olearycrew in https://github.com/projectdiscovery/katana/pull/279
* Updated **regex to detect/extract more endpoints** by @yuzhe-Mortal in https://github.com/projectdiscovery/katana/pull/249
* Updated debug logging by @tarunKoyalwar in https://github.com/projectdiscovery/katana/pull/278
Issues closed in release - https://github.com/projectdiscovery/katana/milestone/4
## New Contributors
* @olearycrew made their first contribution in https://github.com/projectdiscovery/katana/pull/279
* @tarunKoyalwar made their first contribution in https://github.com/projectdiscovery/katana/pull/278
* @yuzhe-Mortal made their first contribution in https://github.com/projectdiscovery/katana/pull/249
* @RamanaReddy0M made their first contribution in https://github.com/projectdiscovery/katana/pull/356
**Full Changelog**: https://github.com/projectdiscovery/katana/compare/v0.0.3...v1.0.0