# Introduction
FarmStack is a reference implementation of an open and interoperable data sharing protocol in agriculture sector.
FarmStack is required because:
* Relevant farmer profile including farmer activity not available
* Lack of trust on misuse or under utilisation of data with a centralised data warehouse
* Need to comply with evolving data policy and privacy safeguarding measures
* Existing data integration tools lack the customisation
## Quick start
```bash
git clone https://github.com/digitalgreenorg/farmstack-open.git
cd farmstack-open
python3 setup.py
```
In browser, open: and follow the instructions
For more details see [local deployment](/deploying-farmstack/local-deployment).
Requirements:
* Python3.6+
* Docker Desktop (for MacOS and Windows with wsl2).
* Docker and Docker Compose for ubuntu
## Deployment and Guides
You can see FarmStack in action by [Setting up Video Library Connector](/deploying-farmstack/setting-up-video-library-connector) for fetching data from [Digital Green's Video Library](https://solutions.digitalgreen.org/videos/library).
Or you can follow one of our step-by-step guide to setup your own connectors:
* [Configuring a connector with CSV file](/operation-guides/creating-a-connector-with-csv)
* [Dockerizing applications to be compatible with FarmStack Connector](/operation-guides/dockerization)
* [Install a dockerized application with a connector](/operation-guides/install-a-nodejs-application-with-connector)
## Features
FarmStack enables network of data providers and consumers through a suite of products and functionalities:
1. Share data directly without any third party through trusted **peer to peer (p2p) connector**.
2. Empower the data provider to restrict usage of data through **usage policies**.
3. Give control of data back to the farmers by managing consent using **data wallet**.
4. Enable entities to create plugins to make their **data discoverable (description of data)**.
FarmStack is the sum total of all the peer to peer connectors and associated usage policies.
For more information visit [our website](https://www.farmstack.co) or [Confluence Page](https://digitalgreenorg.atlassian.net/wiki/spaces/FS/overview).
## Contributing
We would love your contribution to this project no matter big or small.
To get started see our [contribution guide](/contributing-to-farmstack).
## Roadmap
@TODO - Add Roadmap
## License
Farmstack is licensed under Apache License 2.0. See [LICENSE](https://github.com/digitalgreenorg/farmstack-open/tree/576aa2ca789cd6910b3aee1d7ebd9cce08d73cf6/LICENSE/README.md) file for licensing information
# Quick Start
# Deploy Farmstack
## Quick deployment on Ubuntu
* Running this command will install Docker automatically on Ubuntu, if it is not available.
```bash
curl https://raw.githubusercontent.com/digitalgreenorg/farmstack-open/main/scripts/install.sh > fs-install.sh && bash fs-install.sh
```
## Deployment on Ubuntu, MacOS and Windows (with WSL2)
* Install Docker on your system according to OS:
* [Docker Desktop for MacOS and Windows](https://www.docker.com/products/docker-desktop).
* [Docker install on Ubuntu](https://docs.docker.com/engine/install/ubuntu/)
* Clone the repository and open it.
* Run the setup.py file using python3
```bash
git clone https://github.com/digitalgreenorg/farmstack-open.git
cd farmstack-open
python3 setup.py
```
After the setup is done, you can view the connector setup UI at: .
## Next steps
{% content-ref url="/pages/-Mfe3434Q89Btz5X8AmJ" %}
[Setting up Self-Managed Connectors](/deploying-farmstack/setting-up-self-managed-connectors)
{% endcontent-ref %}
{% content-ref url="/pages/-MfdAtPNNbxT1ScTJsaC" %}
[Configuring a connector with CSV](/operation-guides/creating-a-connector-with-csv)
{% endcontent-ref %}
{% content-ref url="/pages/-MfdY-WXSKTGsF45YPVa" %}
[Install a dockerized application with connector](/operation-guides/install-a-nodejs-application-with-connector)
{% endcontent-ref %}
# Removing farmstack
# Deploying Farmstack
# Installing Prerequisites
## Deployment on Linux, MacOS and Windows (with WSL2)
* Install Docker on your system according to OS:
* [Docker Desktop for MacOS and Windows](https://www.docker.com/products/docker-desktop)
* [Docker install on Ubuntu](https://docs.docker.com/engine/install/ubuntu/)
* For Ubuntu, also install docker-compose:
```bash
sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
```
* Install the dependencies on ubuntu:
```bash
sudo apt install python3-pip python3-venv
```
# Setting up Self-Managed Connectors
## Overview
This tutorial will guide you through the process of running an example self managed connector end-to-end setup. Kindly complete the prerequisites before following this guide.
{% content-ref url="/pages/-MfdqmhMoJP6ITTt0LFt" %}
[Installing Prerequisites](/deploying-farmstack/installing-prerequisites)
{% endcontent-ref %}
## Setting up FarmStack connector
In the terminal window type the following command to run FarmStack setup:
```
python3 setup.py
```
After the setup is complete, in the browser window, open the installer frontend by typing .
## Running example connectors
Navigate to New Connector tab
Give the provider a unique name, for example, `Test Provider 1` and click `Next`
Give the consumer a unique name, for example, `Test Consumer 1` and click `Next`
Verify the details and start the connection by clicking `Set up Connection` button.
Depending on your system resources and internet speed this step could take anywhere from a few seconds to a few minutes. You can check the progress in the terminal window.
When the connector setup is complete, open the status tab and click `View transferred data` link to see your data.
Kindly wait a couple of minutes for the contract negotiation process of provider and consumer to complete before they can start sharing the data.
After the negotiation, the provider will start streaming data to the consumer. Kindly refresh to see the data shared into the consumer application.
This completes the tutorial for Setting up a Managed connector with usage control example. If you face any issue while running the self managed connector, kindly open a new issue in the Github repository and our experts will guide you.
# Setting up Video Library Connector
## Overview
This tutorial will guide you through the process of installing and running a Video Library Data Consumer. The video library can be found here: [Digital Green Video Library](https://solutions.digitalgreen.org/videos/library).
Kindly complete the prerequisites before proceeding with the guide.
{% content-ref url="/pages/-MfdqmhMoJP6ITTt0LFt" %}
[Installing Prerequisites](/deploying-farmstack/installing-prerequisites)
{% endcontent-ref %}
## Setting up FarmStack connector
In the terminal window type the following command to run FarmStack setup:
```
python3 setup.py
```
After the setup is complete, in the browser window, open the installer frontend by typing .
## Running the connector
Click on the Add New + button, which will open a dialog to setup connector.
Select the connector you want to run. Here we'll run the Google Sheets connector.
Click on Next which will take you to the configure tab.
In the configure tab, enter a google email you would like to share the data with. You can also change the sheet title here.
Clicking Next will take you to the connect tab.
In the connect tab, you can see the provider connectors available to connect.
Select the Video List Provider (DG - Coco) from the list of providers and click Finish.
On the homepage, the table will update to show the currently running connectors.
From this table you can open the homepage of the connector or delete the connector by clicking he red bin icon.
## Configuring the connectors
### Google Sheets Connector
Open the homepage of the App and click on the Sync Data button to fill the data in Google Sheet.
### CSV Connector
Open the homepage of the App and click on the Sync Data button to generate a csv. Once the CSV file is generated you click on Download option to download the CSV file.
# Local Deployment
{% hint style="info" %}
This deployment has been tested on Linux, MacOS and Windows (with wsl2 and Docker Desktop on wsl2).
{% endhint %}
## Quick deployment on Linux
* Running this command will install Docker automatically on Linux, if it is not available.
```bash
curl https://raw.githubusercontent.com/digitalgreenorg/farmstack-open/main/scripts/install.sh > fs-install.sh && bash fs-install.sh
```
## Deployment on Linux, MacOS and Windows (with WSL2)
* Install Docker on your system according to OS:
* [Docker Desktop for MacOS and Windows](https://www.docker.com/products/docker-desktop)
* [Docker install on Ubuntu](https://docs.docker.com/engine/install/ubuntu/)
* For Ubuntu, also install docker-compose:
```bash
sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
```
* Install the dependencies on ubuntu:
```bash
sudo apt install python3-pip python3-venv
```
* Clone the repository and open it.
* Run the setup.py file using python3
```bash
git clone https://github.com/digitalgreenorg/farmstack-open.git
cd farmstack-open
python3 setup.py
```
After the setup is done, you can view the connector setup UI at: .
# Deployment on AWS EC2
# Guides
# Configuring a connector with CSV
## Overview
This tutorial will describe how to setup FarmStack connector for local csv files. Kindly follow the steps to install FarmStack requirements before proceeding with this setup.
## Local CSV file
You can follow this process for any file, here we will be using a file called cities.csv present in Downloads directory in home folder.
Clone the FarmStack Github repository on your local machine and open it.
```
git clone https://github.com/digitalgreenorg/farmstack-open.git
cd farmstack-open
```
In the FarmStack repository, open `example-provide-routes.yaml` file in `fs-config/usage-control-example/` directory.
```
code fs-configs/usage-control-example/example-provider-routes.xml
```
In the route `sendData`, replace `sample_data1.csv` with filename of your CSV file.
Next, open `docker-compose-provider.yaml` file in `fs-config/usage-control-example/` directory.
```
code fs-configs/usage-control-example/docker-compose-provider.yaml
```
Here comment out the lines which mount `sample_data1.csv` and `sample_data2.csv` to the docker container, and add the line to mount the `cities.csv` to the container, as shown here:
There is no limit on the number of CSV files that can be mounted on connector, follow the same instructions for more csv files.
Save the files and in the terminal window type the following command to run FarmStack setup:
```
python3 setup.py
```
After the setup is complete, in the browser window, open the installer frontend by typing [localhost:8000](http://localhost:8000).
Follow the steps to create your connectors, give a unique name to your connectors such as cities-provider and cities-consumer.
Start the connection by clicking Setup Connection Button.
When the connector setup is done click View transferred data link to see your data. Kindly wait a couple of minutes for the contract negotiation process of provider and consumer to complete before they can start sharing the data.
## Online CSV file
Watch this space for super exciting updates. Our bots are already hard at work to create this page for you.
## Next Steps
This completes the tutorial for CSV file transfer through FarmStack Provider Connector. If you face any issue while setting up your own csv file kindly open a new issue in the github repository and our experts will guide you.
# Configuring a connector with JSON/Rest API
Watch this space for super exciting updates. Our bots are already hard at work using GPT-3 to create this page for you.
# Configuring a connector with Google Sheets
Watch this space for super exciting updates. Our bots are already hard at work using GPT-3 to create this page for you.
# Dockerization
# NodeJS
## Overview
This tutorial will describe how to convert your NodeJS application to docker application for compatibility with FarmStack connector.
## Dockerization Process
Create a new file named `Dockerfile` in the application folder and open in your favorite text editor.
```
touch Dockerfile
```
Copy this code into the `Dockerfile`
First we are going to define the image we are going to use. Here we are using latest alpine image of node to keep the size of NodeJS application small. You can use any image available in the [Node's Dockerhub](https://hub.docker.com/_/node).
```bash
FROM node:alpine
```
Next we will create a directory to copy all our application code inside the image.
```bash
# Create app directory
WORKDIR /app
```
Since we are using node image, `node` and `npm` are already installed in this image. We just need to copy our `package.json` and `package-lock.json` files.
```bash
# Copy package.json and package-lock.json files to image
COPY package*.json ./
RUN npm install
```
Here, we copy `package.json` files before copying complete project. This is done to take advantage of Docker layers caching and only install dependencies if the files have changed. You can find more information about this [here](http://bitjudo.com/blog/2014/03/13/building-efficient-dockerfiles-node-dot-js/).
Now we will copy your app's source code to docker image.
```bash
# Copy app's source code
COPY . .
```
Our `sample-nodejs` app binds to port `8081` so we will map this port to docker `daemon` by using `EXPOSE` command. If your app uses any other port, kindly change it
```bash
EXPOSE 8081
```
In this last step, define the command to start your application. Our sample application starts with`npm start` command. You can also use a shell script file here which executes to start your server.
```bash
ENTRYPOINT ["npm","start"]
```
This should be your final `Dockerfile`
{% code title="Dockerfile" %}
```yaml
FROM node:alpine
# Create app directory
WORKDIR /app
# Copy package.json and package-lock.json files to image
ADD package*.json ./
RUN npm install
# Copy app's source code
COPY . .
EXPOSE 8081
ENTRYPOINT [ "npm", "start" ]
```
{% endcode %}
You can find the final `Dockerfile` [here](https://github.com/digitalgreenorg/farmstack-open/blob/main/tools/dockerfiles/NodeJS.Dockerfile).
### .dockerignore file
Create a `.dockerignore` file in the same directory as your `Dockerfile`. Add the following lines to the file:
{% code title=".dockerignore" %}
```bash
node_modules
npm-debug.log
.npm
```
{% endcode %}
## Next Steps
{% content-ref url="/pages/-MfdY-WXSKTGsF45YPVa" %}
[Install a dockerized application with connector](/operation-guides/install-a-nodejs-application-with-connector)
{% endcontent-ref %}
# Django
# Flask
# Install a dockerized application with connector
## Overview
This tutorial will describe how to run the dockerized application with the connector. This tutorial does not contain information on dockerization of the app. Kindly [dockerization tutorial](/operation-guides/dockerization) according to your app.
You can find the sample-nodejs application used in this tutorial [here](https://github.com/digitalgreenorg/farmstack-open/tree/main/fs-consumer-apps/sample-nodejs).
## Installing dockerized application with connector
Kindly follow the previous tutorial to locally setup and deploy FarmStack before proceeding with this next step, if not done already.
{% content-ref url="/pages/-MfdqmhMoJP6ITTt0LFt" %}
[Installing Prerequisites](/deploying-farmstack/installing-prerequisites)
{% endcontent-ref %}
Clone the FarmStack repository and open in terminal, using following commands:
```bash
git clone https://github.com/digitalgreenorg/farmstack-open.git
cd farmstack-open
```
Open `prepareConsumerApp.sh` in `scripts` folder in your favorite editor.
Edit the following variable to according to your application:
```bash
# Application parameters
# This should be a relative path to the config directory or an absolute path.
CONSUMER_APP_DIR=../../fs-consumer-apps/sample-nodejs
CONSUMER_APP_NAME='sample-nodejs' #change application name
CONSUMER_APP_PORT=8081 #Change port according to your application
SYSTEM_PORT=8081 # Port on the system you want to map to your application's port
```
You can also edit the parameters for the example configuration according to your requirements, but it is advised to leave these variables untouched, unless you know what you're doing.
```bash
# Config parameters
CONFIG_DIR='fs-configs/usage-control-example'
PROVIDER_CONFIG_FILE=example-provider-routes.xml
CONSUMER_DOCKER_COMPOSE=docker-compose-consumer.yaml
```
In the terminal, execute the script from `farmstack-open` directory:
```bash
./scripts/prepareConsumerApp.sh
```
This script will create a docker image for your application and modify the required usage control parameters in the `example-provider-routes.xml` file.
In the terminal window type the following command to run FarmStack setup:
```
python3 setup.py
```
After the setup is complete, in the browser window, open the installer frontend by typing [localhost:8000](http://localhost:8000).
Follow the steps to create your connectors, give a unique name to your connectors such as cities-provider and cities-consumer.
Start the connection by clicking Setup Connection Button.
When the connector setup is done click View transferred data link to see your data. Kindly wait a couple of minutes for the contract negotiation process of provider and consumer to complete before they can start sharing the data.
This completes the tutorial for running the dockerized application with the consumer connector. If you face any issue while running your consumer app kindly open a new issue in the github repository and our experts will guide you.
# Contributing to FarmStack
Thank you for your interest in contributing to FarmStack, currently we are building our contribution guidelines. Meanwhile you can contact us on our [website](https://www.farmstack.co) and [confluence](https://digitalgreenorg.atlassian.net/wiki/spaces/FS/overview).