Allow manual imports of data

[bernhardh]

As discussed in another ticket (#34 (comment)) I am not able to use plaid, but in my eyes, your package would be a great tool, so why not just make it more API agnostic and allow to support multiple different apis (driver based?) and (for me) more important also allow csv imports? Because the filtering, alerts, metrics, etc. doesn't depend on plaid or at least shouldn't, right?

[austinkregel]

I think manual importing is a FANTASTIC idea! 😄

🤔My biggest question would be how should CSV imports be supported? The format the CSV could be in would likely change from bank to bank and country to country 🤔 Custom imports would have to be able to map the CSV to the fields we need and handle missing fields some how.

Or I guess we could try to just make a "standard" format for this project and help people make conversion scripts for their CSVs?

Thoughts? 🤔 😄

[bernhardh]

I have already something in my mind and if I find time, I would try to implement it. But basically it should be based on something like this: https://jgile.github.io/vue-csv-import/

My thoughts - step by step:

1. Decouple the package from plaid (so that it isn't mandatory)
2. Allow to import a csv file with a predefined structure
3. Make a import modal where you can define the structure (like https://jgile.github.io/vue-csv-import/)
4. Allow to save the defined structure so that next time, you don't have to do it again
5. Prevent multiple import of same rows

What do you think?

[austinkregel]

I really like that package 😄 I think it would definitely be useful for uploading transactions 💯

🤔 How should different bank accounts be tracked with manual imports? 🤔

[austinkregel]

For shirts and giggles I went to one of the banks I use and tried to download my transactions and they stopped supporting transaction downloads 😬

I tried my other bank, and I was able to download the transactions, but the info was less than helpful 😅 Here is the exact, unedited file.... (yes, this is the raw file... It just doesn't even have really identifying information 😅 )

Account Designator,Posted Date,Serial Number,"Description",Amount,CR/DR
CHECKING            ,10/08/20,0000000000,"From Share 00 ACH Transfer Deposit: From Share 00",000000005.00,CR
CHECKING            ,09/24/20,0000000000,"From Share 00 ACH Transfer Deposit: From Share 00",000000005.00,CR
CHECKING            ,09/10/20,0000000000,"From Share 00 ACH Transfer Deposit: From Share 00",000000005.00,CR

🤔 If your bank supports a different file format for download I'd love to see an example! (Please remove identifying information, I only care about the format 😉 )

[bernhardh]

Hello,

thinking How should different bank accounts be tracked with manual imports? thinking

of course you would need to manage multiple accounts, but thats not CSV import specific.

yes, this is the raw file

Thats one problem of these csv exports. The data is crap and of course not standardized, so it will not work in a usefull way for some banks. Here are examples of my banks (with anonymized data):

Bank 1 (Austria)

AT1234567890123456789;Bezahlung Karte                              MC/000001234 2320  K001 07.10. 18:35 SHOPNAME 1002\\WIEN\1010      W;07.10.2020;07.10.2020;-4,93;EUR
AT1234567890123456789;111-1111111-1111111 Amazon.de 3ASDASD43434 OG/000003451 TUBDXXXXX DE12345678912345678 AMAZON EU S.A R.L., NIEDERLASSUNG D;20.09.2020;20.09.2020;-25,67;EUR

The csv has no headers, but here is my own interpretation:

My iban	Description	date	booking date	value	currency
AT1234567890123456789	Bezahlung Karte MC/000001234 2320 K001 07.10. 18:35 SHOPNAME 1002\WIEN\1010 W	07.10.2020	07.10.2020	-4,93	EUR
AT1234567890123456789	111-1111111-1111111 Amazon.de 3ASDASD43434 OG/000003451 TUBDXXXXX DE12345678912345678 AMAZON EU S.A R.L., NIEDERLASSUNG D	20.09.2020	20.09.2020	-25,67	EUR

• First row is a regular payment with bank card in a grocery store
• Second row is a payment for a amazon order (including the amazon order number 111-1111111-1111111)

Bank 2 (Germany)

"Datum","Empfänger","Kontonummer","Transaktionstyp","Verwendungszweck","Kategorie","Betrag (EUR)","Betrag (Fremdwährung)","Fremdwährung","Wechselkurs"
"2020-08-10","PayPal Europe S.a.r.l. et Cie S.C.A","FR123456789123456789","Lastschriftumkehrung",". FIVERR COM, Ihr Einkauf bei FIVERR COM","Gutschriften","39.23","","",""

The csv has headers, with the following translation:

"Date","Recipient","Account Number (IBAN)","Transaction type","Reference","Category","Value (EUR)","Value (other currencies)","foreign currency","exchange rate"

General infos:

IBAN = https://en.wikipedia.org/wiki/International_Bank_Account_Number - common bank account number in europe, middle east, north+westafrica, brasil, central america, etc.

[austinkregel]

This is great information 🙇 Thank you 🙇

🤔 I think Something else that would have to happen is a standardization of the amount (Plaid has debit payments as negative values, and credit charges as positive values 🤔 )

🤔 This also gives me more ideas 🤔 (also makes me wish my bank gave me that much info 😞

[austinkregel]

I'll try to hack something together this weekend for this 😄 but I also encourage you to fork the repo and give it a shot too with a PR since it's #hacktober 😄

[bernhardh]

I had now some time to investigate and I have already some ideas how to do it. But right now, I am not sure, how to decouple the system from plaid, since atm the system is hardcoded linked to a plaid account. I tried a bit to play around to remove this hard connection, but found out the hard way, that it would be much easier for someone who knows the code better (you) and especially who can test it with his plaid account, if it still work (I can't)?

This project has basically 4 main features:

1. Sync transactions with your account
2. List transactions
3. Alerts
4. Metrics
5. Budgets

where only the first has to do with plaid. The others are (or should be) datasource agnostic and don't care if data comes from plaid api, csv, finapi.io or whatever.

The ideal solution would be, that you have a User, who can have many Accounts where each Account has its own AccountDriver (Plaid, CSV, finapi.io, tink.com, etc.). So the Accounts itself, doesn't know how it gets the data, this is a task for the AccountDriver.

Would you consider to do this decoupling and move all plaid specific stuff to a separeted AccountDriver (or whatever you want to call it). Then I can make an CSV AccountDriver and others could create their own Drivers for there bank, country, etc?

Btw. since I really like the idea of this project and I was looking for something like this, I would afterwards also be interested in add additional features, translate it to german, help ypu on tickets if you want, etc.)

[austinkregel]

Apologies for my silence on this 😅 I did start something, but ran into the problems you described with Plaid being pretty coupled with the app and then I distracted by American Politics 😓

It's largely kept me away from my projects 😬 (Mostly trying to keep from stressing out)

---

I would absolutely love some help with this project! 😄 Though, as I'm sure you found out, this project is still a bit of a mess from my many iterations of this 😅

Also, I agree that only the syncing of transactions/accounts should be touching any kind of remote service. Everything else should be done via events once the transactions are creating/created.

To make small steps towards supporting other drivers, I'd like to put together a proof of concept (POC) for a slightly more abstract version of the existing code. Still just using Plaid, but allowing for things to be less coupled to just Plaid :) Though, I won't be able to start it until this weekend

If you'd like to refactor stuff in the meantime, please go right ahead 😄 I'm not attached to the code, just the end-result of what the code does :)

[bernhardh]

I hoped you would have maybe a better insight of the code and see, how to easy decouple it from the plaid code.

Please don't get me wrong, I don't want to offend you and what you have done so far is great and if I could use plaid, I would love it, but I am not sure, if it is then wise to work around it. Especially since there are a lot of other problems, missing (micro) features, problems, etc.

So this looks more or less like a rewrite job, then a refactor one and that would be a lot of work. I will have to think about in detail.