dep lib change goproxy->martian #238

tarunKoyalwar · 2023-02-21T21:28:02Z

Proposed changes

replace unmaintained goproxy -> google/martian
bug fixes and more
closes Replace unmaintained goproxy with martian #174

Checklist

Pull request is created against the dev branch
All checks passed (lint, unit/integration/regression tests etc.) with my changes
I have added tests that prove my fix is effective or that my feature works
I have added necessary documentation (if appropriate)

proxy.go

+func (p *Proxy) getRoundTripper() (http.RoundTripper, error) {
+	roundtrip := &http.Transport{
+		TLSClientConfig: &tls.Config{
+			InsecureSkipVerify: true,


proxy.go

+	if len(p.options.UpstreamHTTPProxies) > 0 {
+		roundtrip = &http.Transport{Proxy: func(req *http.Request) (*url.URL, error) {
+			return url.Parse(p.rbhttp.Next())
+		}, TLSClientConfig: &tls.Config{InsecureSkipVerify: true}}


proxy.go

+			socks5Dialer := socks5Dialers[socks5Proxy]
+			// use it to perform the request
+			return socks5Dialer.Dial(network, addr)
+		}, TLSClientConfig: &tls.Config{InsecureSkipVerify: true}}


tarunKoyalwar · 2023-03-06T05:45:04Z

while proxify is good it feels too invasive i.e errors cause domain/website to not render at all or correctly

Setup

proxify -ha "127.0.0.1:8080" -v
install proxify CA in browser and proxy traffic

With `goproxy` lib i.e before this PR

with `martian` lib i.e in this PR

As Shown in above images . there does not seem to be a stable and robust library but in comparison martian has less errors .

What are these errors ?

tls handhshake / other tls failure (only occurs with goproxy) . random io.EOF errors and CONNECT request failures

Anything else

what's wierd is that after these errors we close connection (net.Conn) but browser request is not failed/closed instead is failed after some time due to timeout

tarunKoyalwar · 2023-03-06T05:52:21Z

Errors obtained while debugging

./proxify -ha "127.0.0.1:8080" -v 

                       _ ___    
   ___  _______ __ __ (_) _/_ __
  / _ \/ __/ _ \\ \ // / _/ // /
 / .__/_/  \___/_\_\/_/_/ \_, / 
/_/                      /___/	v0.0.8

		projectdiscovery.io

[INF] HTTP Proxy Listening on 127.0.0.1:8080
[INF] Socks5 Proxy Listening on 127.0.0.1:10080
[INF] Saving proxify traffic to logs
2023/03/06 11:17:50 ERROR: martian mitm: error peeking message through CONNECT tunnel to determine type dns.projectdiscovery.io:443 : EOF
2023/03/06 11:17:50 fragment values is 0
2023/03/06 11:17:50 got connect request but not a tls handshake
2023/03/06 11:17:50 
-------------
last successful parsed req and resp before error was
`<nil>`
`<nil>`
2023/03/06 11:17:50 got `malformed HTTP request "\x00"` data read from raw conn "\x00"
2023/03/06 11:17:50 ERROR: martian: failed to read request: got `malformed HTTP request "\x00"` data read from raw conn "\x00"
2023/03/06 11:17:50 martian: closing connection: 127.0.0.1:52595
2023/03/06 11:17:50 
-------------
last successful parsed req and resp before error was
`&{CONNECT https://snid.snitcher.com:443 HTTP/1.1 1 1 map[Connection:[keep-alive] Proxy-Connection:[keep-alive] User-Agent:[Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/110.0]] {} <nil> 0 [] false snid.snitcher.com:443 map[] map[] <nil> map[] 127.0.0.1:52582 snid.snitcher.com:443 <nil> <nil> <nil> <nil>}`
`<nil>`
2023/03/06 11:17:50 got `EOF` data read from raw conn ""
2023/03/06 11:17:50 martian: closing connection: 127.0.0.1:52582
2023/03/06 11:17:55 
-------------
last successful parsed req and resp before error was
`<nil>`
`<nil>`
2023/03/06 11:17:55 got `EOF` data read from raw conn ""
2023/03/06 11:17:55 martian: closing connection: 127.0.0.1:52600
2023/03/06 11:17:55 
-------------
last successful parsed req and resp before error was
`<nil>`
`<nil>`
2023/03/06 11:17:55 got `EOF` data read from raw conn ""
2023/03/06 11:17:55 martian: closing connection: 127.0.0.1:52588
2

tarunKoyalwar · 2023-03-06T05:58:19Z

Verdict

unlike goproxy , martian lib is simple enough with good/proper implementation (all logic we need is in proxy.go file) . If we can fix these errors we can submit a patch upstream . or embed in proxify / fork and remove all bloated logic

Root Cause of these Errors

the root cause of these errors seems to be reading/handling net.Conn . All above shown errors are caused in interaction between browser <-> proxy

tarunKoyalwar · 2023-03-06T06:05:40Z

@Mzack9999

Debugging and Fixing Errors

Fork of martian with some bug fixes and some modifications for verbosity is available at https://github.com/tarunKoyalwar/martian . clone it
clone the pr branch and replace directory mentioned in go.mod under replace directive with path of above forked martian

Leads

errors originate in martian so we only need to fix/debug martian
All required logic is present in proxy.go file in martian .Possible leads / starting out in martian proxy.go file
- .Serve() -> .handleLoop() -> .handle() -> .readrequest()

Mzack9999

implementation: lgtm
needs a final cross-check if any furher finalization is needed

tarunKoyalwar · 2023-03-29T20:04:24Z

TODO

Blocked until fix malformed chunked encoding martian#3 is resolved
Implement TLS Passthrough flags ,options etc
fix broken pipe errors martian#2

Note: Most of changes in PR are breaking changes

tarunKoyalwar · 2023-03-29T20:06:38Z

@ehsandeep , After some testing it seems like it is almost stable now only malformed chunked encoding is blocker apart from that there weren't issues . (tested with more than 20k live traffic on different sites)

djallalzoldik · 2023-05-13T08:58:56Z

I use the lateast version of proxify but steel have the same error, the certificate not work

dep lib change goproxy->martian

769ff57

tarunKoyalwar linked an issue Feb 21, 2023 that may be closed by this pull request

Replace unmaintained goproxy with martian #174

Closed

tarunKoyalwar marked this pull request as draft February 21, 2023 21:28

bump go to 1.19

207a5ef

github-advanced-security bot found potential problems Feb 21, 2023

View reviewed changes

Mzack9999 self-requested a review February 24, 2023 16:14

debug net.Conn EOF errors

f120a6e

tarunKoyalwar removed the request for review from Mzack9999 March 6, 2023 05:36

tarunKoyalwar requested a review from Mzack9999 March 6, 2023 06:05

Mzack9999 assigned tarunKoyalwar Mar 6, 2023

Mzack9999 added the Type: Enhancement Most issues will probably ask for additions or changes. label Mar 6, 2023

Mzack9999 and others added 7 commits March 27, 2023 04:18

pointing to fork

871213f

bump libs + less verbosity

5c86c73

Merge branch 'dev' into issue-174-martian-proxy

af1a7bf

bump go version

a01b9a3

typo

5235fef

add error handling

502a181

fixing lint issues

5592879

Mzack9999 marked this pull request as ready for review March 29, 2023 15:56

Mzack9999 approved these changes Mar 29, 2023

View reviewed changes

remove unused code

4eafc5d

tarunKoyalwar assigned Mzack9999 Mar 29, 2023

ehsandeep mentioned this pull request Apr 8, 2023

panic: runtime error: invalid memory address or nil pointer dereference #268

Closed

Merge branch 'dev' into issue-174-martian-proxy

c49b4b6

ehsandeep approved these changes Apr 9, 2023

View reviewed changes

ehsandeep merged commit 0a542bf into dev Apr 9, 2023

ehsandeep deleted the issue-174-martian-proxy branch April 9, 2023 07:12

ehsandeep mentioned this pull request May 13, 2023

The certificate not work #286

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dep lib change goproxy->martian #238

dep lib change goproxy->martian #238

tarunKoyalwar commented Feb 21, 2023 •

edited

Loading

tarunKoyalwar commented Mar 6, 2023 •

edited

Loading

tarunKoyalwar commented Mar 6, 2023

tarunKoyalwar commented Mar 6, 2023 •

edited

Loading

tarunKoyalwar commented Mar 6, 2023

Mzack9999 left a comment

tarunKoyalwar commented Mar 29, 2023

tarunKoyalwar commented Mar 29, 2023

djallalzoldik commented May 13, 2023

dep lib change goproxy->martian #238

dep lib change goproxy->martian #238

Conversation

tarunKoyalwar commented Feb 21, 2023 • edited Loading

Proposed changes

Checklist

tarunKoyalwar commented Mar 6, 2023 • edited Loading

Setup

With goproxy lib i.e before this PR

with martian lib i.e in this PR

What are these errors ?

Anything else

tarunKoyalwar commented Mar 6, 2023

Errors obtained while debugging

tarunKoyalwar commented Mar 6, 2023 • edited Loading

Verdict

Root Cause of these Errors

tarunKoyalwar commented Mar 6, 2023

Debugging and Fixing Errors

Leads

Mzack9999 left a comment

Choose a reason for hiding this comment

tarunKoyalwar commented Mar 29, 2023

TODO

tarunKoyalwar commented Mar 29, 2023

djallalzoldik commented May 13, 2023

tarunKoyalwar commented Feb 21, 2023 •

edited

Loading

tarunKoyalwar commented Mar 6, 2023 •

edited

Loading

With `goproxy` lib i.e before this PR

with `martian` lib i.e in this PR

tarunKoyalwar commented Mar 6, 2023 •

edited

Loading