Keeping data private with private data collections on Amazon Managed Blockchain

With the Amazon Managed Blockchain release of Hyperledger Fabric (HLF) version 1.4 support, the private data collections feature is also officially supported. Before HLF 1.2, the channels feature was the primary way to control data privacy amongst a set of members. But channels have significant limitations:

Ordering service nodes still have a copy of all data sent through a channel so that the organizations who run the ordering nodes potentially can get access to a channel data even if their other nodes are not part of that channel.
Data can’t be shared across the channels because a single transaction can’t modify two or more channels’ ledgers at the same time, making an exchange of digital assets harder to implement.

The private data collection feature provides a more granular and secure way to share data amongst a subset of members in the same channel. Unlike with channels, the ordering service receives only a hash of data that was created or modified by chaincode in the private data collection. Also, transactions can modify multiple data items in different private data collections controlled by a single chaincode or multiple chaincodes using chaincode intercommunication.

In this post, we walk through setting up a new channel supporting the private data collections feature and using a sample application to test its data dissemination and data retention capabilities.

Every private data collection has a policy, similar to endorsement policies for chaincode, that specifies which organizations are allowed to get a copy of the data and for how long they can store it. One of the goals was to avoid sending private data to any peers of organizations that weren’t specified in the policy, including the orderers.

To achieve this goal, private data stored in a private data collection is disseminated through a peer-to-peer gossip protocol, while leaving only a hash of a value in the data sent to the orderers. For gossip protocol to work across the organizations, every organization has to specify one or multiple anchor peers in a shared channel configuration. Only then can peers from different organizations discover each other to communicate. Because private data collections and other useful services like service discovery require anchor peers to have already been declared, it’s recommended that all channel members specify anchor peers as part of their initial channel configuration.

Solution overview

The best way to follow this post is to first complete the tutorial Get Started Creating a Hyperledger Fabric Blockchain Network Using Amazon Managed Blockchain and add anchor peers from both organizations to the joint channel “ourchannel”. For instructions, see Add an Anchor Peer to a Channel.

After the channel is joined by the peers from both organizations and anchor peers are configured to facilitate gossip protocol between them, you’re ready to test a sample chaincode with private data collection support. We install and instantiate a sample chaincode called “marbles02_private”, which uses a private data collection to keep private a portion of the data submitted by the user. Then we run some tests to see how private data in the private data collection gets disseminated across the peers and how it gets automatically deleted based on the policy we set up. Let’s get started!

Prerequisites

Before you start, make sure you have Managed Blockchain set up for the Hyperledger Fabric 1.4 network with two members. For instructions, see Get Started Creating a Hyperledger Fabric Blockchain Network Using Amazon Managed Blockchain.

Each member in your network should have the following:

A Hyperledger Fabric client configured on an Amazon Elastic Compute Cloud (Amazon EC2) instance with cloned Hyperledger Fabric samples and the jq tool installed (to install, it run sudo yum install jq ).
At least one peer node.
A joint channel called “ourchannel”, created with enabled features of the latest versions.
Peers of both organizations have joined the joint channel “ourchannel”.
At least one peer from both organizations is configured as an anchor peer on “ourchannel”. For instructions, see Add an Anchor Peer to a Channel.

Testing restricted private data dissemination

The sample chaincode we test allows you to create and transfer digital marbles with public properties like “colour” and “name” and private properties like “price”. For this step, we set different policies in data collection configurations for public and private data collections. Then we query marble data from both collections (called “collectionMarbles” and “collectionMarblePrivateDetails” in this use case) to verify that marble price is available only for one member called Org1.

We use a sample chaincode called “marbles02_private”, which has two data collections: “collectionMarbles”, whose policy allows both members Org1 and Org 2 to keep the data, and “collectionMarblePrivateDetails”, whose policy restricts data dissemination to Org1 only.

On the Org1 EC2 client instance, modify “fabric_samples/chaincode/marbles02_private/collections_config.json” to include both member IDs into the public policy and only member ID of Org1 into the private data collection policy (see the following code). Additionally, set the value of “blockToLive” to “5” to instruct all peers to delete the data after five more blocks are created in the channel. We test that feature in a later step.

[
            {
                "name": "collectionMarbles",
                "policy": "OR('<AMB_MEMBER_ID_FOR_ORG1>.member', '<AMB_MEMBER_ID_FOR_ORG2>.member')",
                "requiredPeerCount": 0,
                "maxPeerCount": 3,
                "blockToLive":1000000,
                "memberOnlyRead": true
            },
            {
                "name": "collectionMarblePrivateDetails",
                "policy": "OR('<AMB_MEMBER_ID_FOR_ORG1>.member')",
                "requiredPeerCount": 0,
                "maxPeerCount": 3,
                "blockToLive": 5,
                "memberOnlyRead": true
            }
        ]

On both Org1 and Org2 EC2 client instances, install the sample chaincode with private data collection support on peers:
```
docker exec cli peer chaincode install -n marblespriv1 -v 1.0 -p github.com/marbles02_private/go/
```

On the Org1 EC2 client instance, instantiate the chaincode with private data collection support:

docker exec cli peer chaincode instantiate -o $ORDERER -C ourchannel -n marblespriv1 -v 1.0 -c '{"Args":["init"]}' --cafile /opt/home/managedblockchain-tls-chain.pem --tls --collections-config /opt/gopath/src/github.com/marbles02_private/collections_config.json -P "OR ('<AMB_MEMBER_ID_FOR_ORG1>.member', '<AMB_MEMBER_ID_FOR_ORG2>.member')"

On the Org1 EC2 client instance, create a new marble in the private data collection, using only a peer of Org1 for endorsement:

export MARBLE=$(echo -n "{\"name\":\"marble1\",\"color\":\"blue\",\"size\":35,\"owner\":\"tom\",\"price\":99}" | base64 | tr -d \\n)

docker exec cli peer chaincode invoke -C ourchannel -n marblespriv1 -c '{"Args":["initMarble"]}' --peerAddresses <PEER_ORG1_DNS_NAME>:<PEER_ORG1_PORT_NUMBER> --tlsRootCertFiles /opt/home/managedblockchain-tls-chain.pem --transient "{\"marble\":\"$MARBLE\"}" -o $ORDERER --cafile /opt/home/managedblockchain-tls-chain.pem –tls

Both Org1 and Org2 should be able to read non-private data of the collection. See the following command:

docker exec cli peer chaincode query -C ourchannel -n marblespriv1 -c '{"Args":["readMarble","marble1"]}'

The private data collection policy collectionMarblePrivateDetails allows only Org1 to have access to private properties of a marble, such as its price. The following command querying the marble’s private properties works on the Org1 client and doesn’t work on the Org2 client:
```
docker exec cli peer chaincode query -C ourchannel -n marblespriv1 -c '{"Args":["readMarblePrivateDetails","marble1"]}'
```

On Org1, you should get private marbles data like the following code:
```
{"docType":"marblePrivateDetails","name":"marble1","price":99}
```

On Org2, you get an error similar to the following message:

Error: endorsement failure during query. response: status:500 message:"{\"Error\":\"Failed to get private details for marble1: GET_STATE failed: transaction ID: e5dd7a53c86fc0cae54b29cb28f298b724cef73d298d724bb507beb385469c84: tx creator does not have read access permission on privatedata in chaincodeName:marblespriv1 collectionName: collectionMarblePrivateDetails\"}"

Testing expansion of private data dissemination to the second organization

Now let’s test what happens if we add Org2 into our policy for “collectionMarblePrivateDetails” collection of our chaincode. After editing the data collections configuration file, we install a new version for our chaincode and upgrade it with a new config.

On the Org1 EC2 client instance, modify “fabric_samples/chaincode/marbles02_private/collections_config.json” to add Org2 to the private collection:

[
            {
                "name": "collectionMarbles",
                "policy": "OR('['<AMB_MEMBER_ID_FOR_ORG1>.member', '<AMB_MEMBER_ID_FOR_ORG2>.member')",
                "requiredPeerCount": 0,
                "maxPeerCount": 3,
                "blockToLive":1000000,
                "memberOnlyRead": true
            },
            {
                "name": "collectionMarblePrivateDetails",
                "policy": "OR('<AMB_MEMBER_ID_FOR_ORG1>.member', '<AMB_MEMBER_ID_FOR_ORG2>.member')",
                "requiredPeerCount": 0,
                "maxPeerCount": 3,
                "blockToLive":5,
                "memberOnlyRead": true
            }
        ]

On both Org1 and Org2 EC2 client instances, install a new version of the sample chaincode with private data collection support on peers:
```
docker exec cli peer chaincode install -n marblespriv1 -v 1.1 -p github.com/marbles02_private/go/
```

On the Org1 EC2 client instance, instantiate the chaincode with private data collection support:

docker exec cli peer chaincode upgrade -o $ORDERER -C ourchannel -n marblespriv1 -v 1.1 -c '{"Args":["init"]}' --cafile /opt/home/managedblockchain-tls-chain.pem --tls --collections-config /opt/gopath/src/github.com/marbles02_private/collections_config.json -P "OR ('<AMB_MEMBER_ID_FOR_ORG1>.member','<AMB_MEMBER_ID_FOR_ORG2>.member')"

On the Org1 EC2 client instance, create a new marble in the private data collection, using only a peer of Org1 for endorsement:

export MARBLE=$(echo -n "{\"name\":\"marble2\",\"color\":\"blue\",\"size\":35,\"owner\":\"tom\",\"price\":99}" | base64 | tr -d \\n)

docker exec cli peer chaincode invoke -C ourchannel -n marblespriv1 -c '{"Args":["initMarble"]}' --peerAddresses <PEER_ORG1_DNS_NAME>:<PEER_ORG1_PORT_NUMBER> --tlsRootCertFiles /opt/home/managedblockchain-tls-chain.pem --transient "{\"marble\":\"$MARBLE\"}" -o $ORDERER --cafile /opt/home/managedblockchain-tls-chain.pem –tls

We should be able to retrieve private data of “marble2” on both Org1 and Org2 now because of the upgraded collection policy. See the following code:
```
docker exec cli peer chaincode query -C ourchannel -n marblespriv1 -c '{"Args":["readMarblePrivateDetails","marble2"]}'
```

We should also be able to retrieve private data of “marble1” on Org2 because of the upgraded collection policy. See the following code:
```
docker exec cli peer chaincode query -C ourchannel -n marblespriv1 -c '{"Args":["readMarblePrivateDetails","marble1"]}'
```

Testing that early records are removed from the private data collection

In addition to keeping the marble price property private, we want to make sure it’s eventually deleted. That way we allow marble holders to create limited sales offers for small group of organizations. To achieve that, we set the “blockToLive” parameter of the private data collection config to “5” in an earlier step, so that after five blocks, the price is “forgotten” by all the peers. That way, the owner is assured that a special sales price is set for a limited period only.

On the Org1 EC2 client instance, create three more marbles in the private data collection, using only a peer of Org1 for endorsement:

for i in $(seq 3 6); do 
	export MARBLE=$(echo -n "{\"name\":\"marble${i}\",\"color\":\"blue\",\"size\":35,\"owner\":\"tom\",\"price\":99}" | base64 | tr -d \\n)

	docker exec cli peer chaincode invoke -C ourchannel -n marblespriv1 -c '{"Args":["initMarble"]}' --peerAddresses <PEER_ORG1_DNS_NAME>:<PEER_ORG1_PORT_NUMBER>  --tlsRootCertFiles /opt/home/managedblockchain-tls-chain.pem --transient "{\"marble\":\"$MARBLE\"}" -o $ORDERER --cafile /opt/home/managedblockchain-tls-chain.pem --tls
	# Sleeping few seconds to ensure we create new block per transaction	
	sleep 3s done

The following command should work on both Org1 and Org2 EC2 client instances:

docker exec cli peer chaincode query -C ourchannel -n marblespriv1 -c '{"Args":["readMarblePrivateDetails","marble4"]}'

Because we added five blocks since marble1 was created, both Org1 and Org2 should automatically delete marble1 according to the policy. Therefore, we shouldn’t be able to retrieve it on Org1 or Org2 EC2 client instances. See the following code:
```
docker exec cli peer chaincode query -C ourchannel -n marblespriv1 -c
'{"Args":["readMarblePrivateDetails","marble1"]}'
```
You should get the following error message:

Error: endorsement failure during query. response: status:500 message:"{\"Error\":\"Marble private details does not exist: marble1\"}"

Conclusion

In this post, we learned how to control data dissemination and retention with private data collections, and the role of anchor peers in this process. We configured a private data collection policy for a sample chaincode and installed it on both peers of two different organizations and instantiated on the joint channel. Then we created a marble with private data initially available only for one organization and made it available to a second organization by changing the private data collection policy and redeploying the sample chaincode. Then we generated more transactions and tested that according to the policy, after five blocks, private data that needs to be forgotten is deleted and no longer available on peers.

For more information about private data collection, see the following:

We hope you found this post useful; happy building on AWS!

About the Author

Nikolay Vlasov is a Senior Blockchain Architect with AWS Professional Services. Being part of a Global Speciality Practice team, he helps clients worldwide to turn their ideas into Pilots, Minimally Viable Products, and Production-ready systems based on blockchain technology.

Gitesh Tyagi is a Senior Software Development Engineer with Amazon Managed Blockchain at AWS based in Seattle.

AWS Database Blog