# Notebook from Bhuvan-21/PySyft
Path: examples/secure-multi-party-computation/Duet/1-DS-1-DO/POC-MPCTensor-Duet-Party2-DO.ipynb

# MPC Tensor - Party 2
### With Duet_____no_output_____In this tutorial we will show you how to perform secure multiparty computation with data you cannot see. There are two parts/notebooks:
* [POC-MPCTensor-Duet-Party1-DS](POC-MPCTensor-Duet-Party1-DS.ipynb). The data-scientist will be the responsible of perform any secure computation.
* [POC-MPCTensor-Duet-Party2-DO](POC-MPCTensor-Duet-Party2-DO.ipynb) (this notebook). The Data Owner will store data in his Duet server and will be available for the data-scientist._____no_output_____## 0 - Libraries
_____no_output_____Let's import the main libraries_____no_output_____

import torch  #  tensor computation
import syft as sy  # core library for remote execution

sy.load("sympc")  # openmined library which helps to perform mpc (see https://github.com/OpenMined/SyMPC)_____no_output_____

## 1 - Duet Server and connection to Data Owner (Party 2)_____no_output_____### 1.1 - Join to Data Scientist Duet Server (Party 1)_____no_output_____

duet_p1 = sy.join_duet("003b50cedc695ad9ae18c3cd6b3ad80a")_____no_output_____

If you see a green message saying CONNECTED!, that means that everything was ok, however there is no data to work with.
Now, the Data Owner wiill launch his own Duet server and will publish some data_____no_output_____## 1.2 - Launch a Duet Server_____no_output_____

duet_p2 = sy.launch_duet()_____no_output_____

Go now to [POC-MPCTensor-Duet-Party1-DS](POC-MPCTensor-Duet-Party1-DS.ipynb), execute Section 1.2 where the data scientist connect with the data owner, and continue with this notebook until the end._____no_output_____## 2- Secret data_____no_output_____Now the Data Owner will publish some data that he wants to share in a private way_____no_output_____Let's check that there is no data at this moment in the server,_____no_output_____

duet_p2.store.pandas_____no_output_____

and let's publish the datasets_____no_output_____### 2.1 - Publish the secret data_____no_output_____

x = torch.tensor([50])
x.send(duet_p2, pointable=True, tags=["#test_tensor_1"], description= "Dummy tensor to test the sum of tensors")_____no_output_____x = torch.tensor([[1,2],[3,4]])
x.send(duet_p2, pointable=True, tags=["#test_tensor_2"], description= "Dummy tensor to test the matmul of tensors")_____no_output_____

Let's check again the stored data_____no_output_____

duet_p2.store.pandas_____no_output_____

If everything was ok, you should see a pandas dataframe with two rows corresponding to the stored datasets. Now the data-scientist has everything ready to perform remote operations via SMPC._____no_output_____### 2.2 -  Automate request response_____no_output_____Now the data owner instead of handling the requests manually, he will accept the request of type `reconstruct` automatically._____no_output_____

duet_p2.requests.add_handler(
    name="reconstruct",
    action="accept"
)_____no_output_____

## Congratulations!!! - Time to Join the Community!

Congratulations on completing this notebook tutorial! If you enjoyed this and would like to join the movement toward privacy preserving, decentralized ownership of AI and the AI supply chain (data), you can do so in the following ways!

### Star PySyft and SyMPC on GitHub
The easiest way to help our community is just by starring the GitHub repos! This helps raise awareness of the cool tools we're building.

* [Star PySyft](https://github.com/OpenMined/PySyft)
* [Star SyMPC](https://github.com/OpenMined/SyMPC/)

### Join our Slack!
The best way to keep up to date on the latest advancements is to join our community! You can do so by filling out the form at http://slack.openmined.org

### Join a Code Project!
The best way to contribute to our community is to become a code contributor! At any time you can go to PySyft GitHub Issues page and filter for "Projects". This will show you all the top level Tickets giving an overview of what projects you can join! If you don't want to join a project, but you would like to do a bit of coding, you can also look for more "one off" mini-projects by searching for GitHub issues marked "good first issue".

* [PySyft Good First Issue Tickets](https://github.com/OpenMined/PySyft/labels/Good%20first%20issue%20%3Amortar_board%3A)
* [SyMPC Good First Issue Tickets](https://github.com/OpenMined/SyMPC/labels/good%20first%20issue)

### Donate
If you don't have time to contribute to our codebase, but would still like to lend support, you can also become a Backer on our Open Collective. All donations go toward our web hosting and other community expenses such as hackathons and meetups!

* [OpenMined's Open Collective Page](https://opencollective.com/openmined)_____no_output_____


# Notebook from PatrickvanZalm/alphapept
Path: nbs/03_fasta.ipynb

# default_exp fasta_____no_output_____

# FASTA

> Functions related to generating spectra from FASTA files_____no_output_____This notebook contains all functions related to creating spectra from FASTA files. In brief, what we are doing is the following:

1. Read a FASTA file and in silico digest the proteins to generate peptides
2. For each peptide, calculate a theoretical spectrum and precursor mass
3. Save spectra

Currently, `numba` has only limited string support. A lot of the functions are therefore Python-native._____no_output_____

#hide
from nbdev.showdoc import *_____no_output_____

## Cleaving

We use regular expressions to find potential cleavage sites for cleaving and write the wrapper `cleave_sequence` to use it._____no_output_____

#export
from alphapept import constants
import re

def get_missed_cleavages(sequences:list, n_missed_cleavages:int) -> list:
    """
    Combine cleaved sequences to get sequences with missed cleavages
    Args:
        seqeuences (list of str): the list of cleaved sequences, no missed cleavages are there.
        n_missed_cleavages (int): the number of miss cleavage sites.
    Returns:
        list (of str): the sequences with missed cleavages.
    """
    missed = []
    for k in range(len(sequences)-n_missed_cleavages):
        missed.append(''.join(sequences[k-1:k+n_missed_cleavages]))

    return missed

def cleave_sequence(
    sequence:str="",
    n_missed_cleavages:int=0,
    protease:str="trypsin",
    pep_length_min:int=6,
    pep_length_max:int=65,
    **kwargs
)->list:
    """
    Cleave a sequence with a given protease. Filters to have a minimum and maximum length.
    Args:
        sequence (str): the given (protein) sequence.
        n_missed_cleavages (int): the number of max missed cleavages.
        protease (str): the protease/enzyme name, the regular expression can be found in alphapept.constants.protease_dict.
        pep_length_min (int): min peptide length.
        pep_length_max (int): max peptide length.
    Returns:
        list (of str): cleaved peptide sequences with missed cleavages.
    """

    proteases = constants.protease_dict
    pattern = proteases[protease]

    p = re.compile(pattern)

    cutpos = [m.start()+1 for m in p.finditer(sequence)]
    cutpos.insert(0,0)
    cutpos.append(len(sequence))

    base_sequences = [sequence[cutpos[i]:cutpos[i+1]] for i in range(len(cutpos)-1)]

    sequences = base_sequences.copy()

    for i in range(1, n_missed_cleavages+1):
        sequences.extend(get_missed_cleavages(base_sequences, i))

    sequences = [_ for _ in sequences if len(_)>=pep_length_min and len(_)<=pep_length_max]

    return sequences_____no_output_____protease = "trypsin"
n_missed_cleavages = 0
pep_length_min, pep_length_max = 6, 65

cleave_sequence('ABCDEFGHIJKLMNOPQRST', n_missed_cleavages, protease, pep_length_min, pep_length_max)_____no_output_____#hide
def test_cleave_sequence():

    protease = "trypsin"
    pep_length_min, pep_length_max = 6, 65

    assert set(cleave_sequence('ABCDEFGHIJKLMNOPQRST', 0, protease, pep_length_min, pep_length_max)) == set(['ABCDEFGHIJK', 'LMNOPQR'])
    assert set(cleave_sequence('ABCDEFGHIJKLMNOPQRST', 1, protease, pep_length_min, pep_length_max)) == set(['ABCDEFGHIJK', 'LMNOPQR', 'ABCDEFGHIJKLMNOPQR'])

test_cleave_sequence()_____no_output_____

## Counting missed and internal cleavages
The following are helper functions to retrieve the number of missed cleavages and internal cleavage sites for each sequence._____no_output_____

#export
import re
from alphapept import constants

def count_missed_cleavages(sequence:str="", protease:str="trypsin", **kwargs) -> int:
    """
    Counts the number of missed cleavages for a given sequence and protease
    Args:
        sequence (str): the given (peptide) sequence.
        protease (str): the protease/enzyme name, the regular expression can be found in alphapept.constants.protease_dict.
    Returns:
        int: the number of miss cleavages
    """
    proteases = constants.protease_dict
    protease = proteases[protease]
    p = re.compile(protease)
    n_missed = len(p.findall(sequence))
    return n_missed

def count_internal_cleavages(sequence:str="", protease:str="trypsin", **kwargs) -> int:
    """
    Counts the number of internal cleavage sites for a given sequence and protease
    Args:
        sequence (str): the given (peptide) sequence.
        protease (str): the protease/enzyme name, the regular expression can be found in alphapept.constants.protease_dict.
    Returns:
        int (0 or 1): if the sequence is from internal cleavage.
    """
    proteases = constants.protease_dict
    protease = proteases[protease]
    match = re.search(protease,sequence[-1]+'_')
    if match:
        n_internal = 0
    else:
        n_internal = 1
    return n_internal_____no_output_____protease = "trypsin"
print(count_missed_cleavages('ABCDEFGHIJKLMNOPQRST', protease))

protease = "trypsin"
print(count_internal_cleavages('ABCDEFGHIJKLMNOPQRST', protease))2
1
#hide
def test_get_missed_cleavages():
    assert count_missed_cleavages('ABCDEFGHIJKLMNOPQRST', 'trypsin') == 2
    assert count_missed_cleavages('ABCDEFGHIJKLMNOPQRST', 'clostripain') == 1

test_get_missed_cleavages()

def test_get_internal_cleavages():
    assert count_internal_cleavages('ABCDEFGHIJKLMNOPQRST', 'trypsin') == 1
    assert count_internal_cleavages('ABCDEFGHIJKLMNOPQRSTK', 'trypsin') == 0

test_get_internal_cleavages()_____no_output_____

## Parsing

Peptides are composed out of amino acids that are written in capital letters - `PEPTIDE`. To distinguish modifications, they are written in lowercase such as `PEPTIoxDE` and can be of arbitrary length. For a modified amino acid (AA), the modification precedes the letter of the amino acid. Decoys are indicated with an underscore. Therefore, the `parse` function splits after `_`. When parsing, the peptide string is converted into a `numba`-compatible list, like so: `PEPoxTIDE` -> `[P, E, P, oxT, I, D, E]`. This allows that we can use the `mass_dict` from `alphapept.constants` to directly determine the masses for the corresponding amino acids._____no_output_____

#export
from numba import njit
from numba.typed import List

@njit
def parse(peptide:str)->List:
    """
    Parser to parse peptide strings
    Args:
        peptide (str): modified peptide sequence.
    Return:
        List (numba.typed.List): a list of animo acids and modified amono acids
    """
    if "_" in peptide:
        peptide = peptide.split("_")[0]
    parsed = List()
    string = ""

    for i in peptide:
        string += i
        if i.isupper():
            parsed.append(string)
            string = ""

    return parsed

def list_to_numba(a_list) -> List:
    """
    Convert Python list to numba.typed.List
    Args:
        a_list (list): Python list.
    Return:
        List (numba.typed.List): Numba typed list.
    """
    numba_list = List()

    for element in a_list:
        numba_list.append(element)

    return numba_list_____no_output_____print(parse('PEPTIDE'))
print(parse('PEPoxTIDE'))[P, E, P, T, I, D, E]
[P, E, P, oxT, I, D, E]
#hide
def test_parse():
    peptide = "PEPTIDE"
    assert parse(peptide) == list_to_numba(["P", "E", "P", "T", "I", "D", "E"])
    peptide = "PEPoxTIDE"
    assert parse(peptide) == list_to_numba(["P", "E", "P", "oxT", "I", "D", "E"])
    peptide = "PEPTIDE_decoy"
    assert parse(peptide) == list_to_numba(["P", "E", "P", "T", "I", "D", "E"])

test_parse()_____no_output_____

## Decoy

The decoy strategy employed is a pseudo-reversal of the peptide sequence, keeping only the terminal amino acid and reversing the rest. Additionally, we can call the functions `swap_KR` and and `swap_AL` that will swap the respective AAs. The function `swap_KR` will only swap terminal AAs. The swapping functions only work if the AA is not modified._____no_output_____

#export
@njit
def get_decoy_sequence(peptide:str, pseudo_reverse:bool=False, AL_swap:bool=False, KR_swap:bool = False)->str:
    """
    Reverses a sequence and adds the '_decoy' tag.
    Args:
        peptide (str): modified peptide to be reversed.
        pseudo_reverse (bool): If True, reverse the peptide bug keep the C-terminal amino acid; otherwise reverse the whole peptide. (Default: False)
        AL_swap (bool): replace A with L, and vice versa. (Default: False)
        KR_swap (bool): replace K with R at the C-terminal, and vice versa. (Default: False)
    Returns:
        str: reversed peptide ending with the '_decoy' tag.
    """
    pep = parse(peptide)
    if pseudo_reverse:
        rev_pep = pep[:-1][::-1]
        rev_pep.append(pep[-1])
    else:
        rev_pep = pep[::-1]

    if AL_swap:
        rev_pep = swap_AL(rev_pep)

    if KR_swap:
        rev_pep = swap_KR(rev_pep)

    rev_pep = "".join(rev_pep)

    return rev_pep

@njit
def swap_KR(peptide:str)->str:
    """
    Swaps a terminal K or R. Note: Only if AA is not modified.

    Args:
        peptide (str): peptide.

    Returns:
        str: peptide with swapped KRs.
    """
    if peptide[-1] == 'K':
        peptide[-1] = 'R'
    elif peptide[-1] == 'R':
        peptide[-1] = 'K'

    return peptide

@njit
def swap_AL(peptide:str)->str:
    """
    Swaps a A with L. Note: Only if AA is not modified.
    Args:
        peptide (str): peptide.

    Returns:
        str: peptide with swapped ALs.
    """
    i = 0
    while i < len(range(len(peptide) - 1)):
        if peptide[i] == "A":
            peptide[i] = peptide[i + 1]
            peptide[i + 1] = "A"
            i += 1
        elif peptide[i] == "L":
            peptide[i] = peptide[i + 1]
            peptide[i + 1] = "L"
            i += 1
        i += 1

    return peptide

def get_decoys(peptide_list, pseudo_reverse=False, AL_swap=False, KR_swap = False, **kwargs)->list:
    """
    Wrapper to get decoys for lists of peptides
    Args:
        peptide_list (list): the list of peptides to be reversed.
        pseudo_reverse (bool): If True, reverse the peptide bug keep the C-terminal amino acid; otherwise reverse the whole peptide. (Default: False)
        AL_swap (bool): replace A with L, and vice versa. (Default: False)
        KR_swap (bool): replace K with R at the C-terminal, and vice versa. (Default: False)
    Returns:
        list (of str): a list of decoy peptides
    """
    decoys = []
    decoys.extend([get_decoy_sequence(peptide, pseudo_reverse, AL_swap, KR_swap) for peptide in peptide_list])
    return decoys

def add_decoy_tag(peptides):
    """
    Adds a '_decoy' tag to a list of peptides
    """
    return [peptide + "_decoy" for peptide in peptides]_____no_output_____print(swap_AL(parse('KKKALKKK')))
print(swap_KR(parse('AAAKRAAA')))[K, K, K, L, A, K, K, K]
[A, A, A, K, R, A, A, A]
print(get_decoy_sequence('PEPTIDE'))EDITPEP
print(get_decoys(['ABC','DEF','GHI']))['CBA', 'FED', 'IHG']
#hide

def test_swap_AL():
    assert swap_AL(parse("ABCDEF")) == parse("BACDEF")
    assert swap_AL(parse("GHIKLM")) == parse("GHIKML")
    assert swap_AL(parse("FEDCBA")) == parse("FEDCBA")
    assert swap_AL(parse("GHIKL")) == parse("GHIKL")
    assert swap_AL(parse("ABCDEFGHIKLM")) == parse("BACDEFGHIKML")
    assert swap_AL(parse("BBAcCD")) == parse("BBcCAD")
    assert swap_AL(parse("FEDCBA")) == parse("FEDCBA")

test_swap_AL()

def test_swapKR():
    assert swap_KR(parse("ABCDEK")) == parse("ABCDER")
    assert swap_KR(parse("ABCDER")) == parse("ABCDEK")
    assert swap_KR(parse("ABCDEF")) == parse("ABCDEF")
    assert swap_KR(parse("KABCDEF")) == parse("KABCDEF")
    assert swap_KR(parse("KABCRDEF")) == parse("KABCRDEF")
    assert swap_KR(parse("KABCKDEF")) == parse("KABCKDEF")

test_swapKR()

def test_get_decoy_sequence():
    peptide = "PEPTIDER"
    assert get_decoy_sequence(peptide, pseudo_reverse=True) == "EDITPEPR"
    assert get_decoy_sequence(peptide) == "REDITPEP"
    assert get_decoy_sequence(peptide, KR_swap=True, pseudo_reverse=True) == "EDITPEPK"

test_get_decoy_sequence()_____no_output_____

## Modifications

To add modifications to the peptides, we distinguish fixed and variable modifications. Additionally, we make a distinction between whether the modification is only terminal or not.

### Fixed Modifications
Fixed modifications are implemented by passing a list with modified AAs that should be replaced. As a AA is only one letter, the remainder is the modification. _____no_output_____

#export
def add_fixed_mods(seqs:list, mods_fixed:list, **kwargs)->list:
    """
    Adds fixed modifications to sequences.
    Args:
        seqs (list of str): sequences to add fixed modifications
        mods_fixed (list of str): the string list of fixed modifications. Each modification string must be in lower case, except for that the last letter must be the modified amino acid (e.g. oxidation on M should be oxM).
    Returns:
        list (of str): the list of the modified sequences. 'ABCDEF' with fixed mod 'cC' will be 'ABcCDEF'.
    """
    if not mods_fixed:
        return seqs
    else:
        for mod_aa in mods_fixed:
            seqs = [seq.replace(mod_aa[-1], mod_aa) for seq in seqs]
        return seqs_____no_output_____mods_fixed = ['cC','bB']
peptide_list = ['ABCDEF']

add_fixed_mods(peptide_list, mods_fixed)_____no_output_____mods_fixed = ['aA','cC','bB']
peptide_list = ['cABCDEF']

add_fixed_mods(peptide_list, mods_fixed)_____no_output_____#hide
def test_add_fixed_mods():
    mods_fixed = ['cC']
    peptide_list = ['ABCDEF']

    peptides_new = add_fixed_mods(peptide_list, [])
    assert peptides_new == peptide_list

    peptides_new = add_fixed_mods(peptide_list, mods_fixed)
    assert peptides_new == ['ABcCDEF']

test_add_fixed_mods()_____no_output_____

### Variable Modifications

To employ variable modifications, we loop through each variable modification and each position of the peptide and add them to the peptide list. For each iteration in get_isoforms, one more variable modification will be added._____no_output_____

#export
def add_variable_mod(peps:list, mods_variable_dict:dict)->list:
    """
    Function to add variable modification to a list of peptides.
    Args:
        peps (list): List of peptides.
        mods_variable_dict (dict): Dicitionary with modifications. The key is AA, and value is the modified form (e.g. oxM).
    Returns:
        list : the list of peptide forms for the given peptide.
    """
    peptides = []
    for pep_ in peps:
        pep, min_idx = pep_
        for mod in mods_variable_dict:
            for i in range(len(pep)):
                if i >= min_idx:
                    c = pep[i]
                    if c == mod:
                        peptides.append((pep[:i]+[mods_variable_dict[c]]+pep[i+1:], i))
    return peptides

def get_isoforms(mods_variable_dict:dict, peptide:str, isoforms_max:int, n_modifications_max:int=None)->list:
    """
    Function to generate modified forms (with variable modifications) for a given peptide - returns a list of modified forms.
    The original sequence is included in the list
    Args:
        mods_variable_dict (dict): Dicitionary with modifications. The key is AA, and value is the modified form (e.g. oxM).
        peptide (str): the peptide sequence to generate modified forms.
        isoforms_max (int): max number of modified forms to generate per peptide.
        n_modifications_max (int, optional): max number of variable modifications per peptide.
    Returns:
        list (of str): the list of peptide forms for the given peptide
    """
    pep = list(parse(peptide))

    peptides = [pep]
    new_peps = [(pep, 0)]

    iteration = 0
    while len(peptides) < isoforms_max:

        if n_modifications_max:
            if iteration >= n_modifications_max:
                break

        new_peps = add_variable_mod(new_peps, mods_variable_dict)

        if len(new_peps) == 0:
            break
        if len(new_peps) > 1:
            if new_peps[0][0] == new_peps[1][0]:
                new_peps = new_peps[0:1]

        for _ in new_peps:
            if len(peptides) < isoforms_max:
                peptides.append(_[0])

        iteration +=1

    peptides = [''.join(_) for _ in peptides]

    return peptides_____no_output_____mods_variable_dict = {'S':'pS','P':'pP','M':'oxM'}
isoforms_max = 1024

print(get_isoforms(mods_variable_dict, 'PEPTIDE', isoforms_max))
print(get_isoforms(mods_variable_dict, 'AMAMA', isoforms_max))
print(get_isoforms(mods_variable_dict, 'AMAMA', isoforms_max, n_modifications_max=1))['PEPTIDE', 'pPEPTIDE', 'PEpPTIDE', 'pPEpPTIDE']
['AMAMA', 'AoxMAMA', 'AMAoxMA', 'AoxMAoxMA']
['AMAMA', 'AoxMAMA', 'AMAoxMA']

Lastly, we define the wrapper `add_variable_mods` so that the functions can be called for lists of peptides and a list of variable modifications._____no_output_____

#hide
def test_get_isoforms():

    mods_variable_dict = {'S':'pS','P':'pP'}
    peptide = 'PEPTIDE'
    isoforms_max = 1024
    get_isoforms(mods_variable_dict, peptide, isoforms_max)

    assert len(get_isoforms(mods_variable_dict, peptide, isoforms_max)) == 4

test_get_isoforms()_____no_output_____#export
from itertools import chain

def add_variable_mods(peptide_list:list, mods_variable:list, isoforms_max:int, n_modifications_max:int, **kwargs)->list:
    """
    Add variable modifications to the peptide list
    Args:
        peptide_list (list of str): peptide list.
        mods_variable (list of str): modification list.
        isoforms_max (int): max number of modified forms per peptide sequence.
        n_modifications_max (int): max number of variable modifications per peptide.
    Returns:
        list (of str): list of modified sequences for the given peptide list.
    """

    #the peptide_list originates from one peptide already -> limit isoforms here

    max_ = isoforms_max - len(peptide_list) + 1

    if max_ < 0:
        max_ = 0

    if not mods_variable:
        return peptide_list
    else:
        mods_variable_r = {}
        for _ in mods_variable:
            mods_variable_r[_[-1]] = _

        peptide_list = [get_isoforms(mods_variable_r, peptide, max_, n_modifications_max) for peptide in peptide_list]
        return list(chain.from_iterable(peptide_list))_____no_output_____peptide_list = ['AMA', 'AAC']
mods_variable = ['oxM','amC']
isoforms_max = 1024
n_modifications_max = 10

add_variable_mods(peptide_list, mods_variable, isoforms_max, n_modifications_max)_____no_output_____#hide
def test_add_variable_mods():
    mods_variable = ['oxM']
    peptide = ['AMAMA']

    peptides_new = add_variable_mods(peptide, [], 1024, None)
    assert peptides_new == peptide

    peptides_new = add_variable_mods(peptide, mods_variable, 1024, None)

    assert set(['AMAMA', 'AMAoxMA', 'AoxMAMA', 'AoxMAoxMA']) == set(peptides_new)

    # Check if number of isoforms is correct
    peptides_new = add_variable_mods(peptide, mods_variable, 3, None)
    assert len(peptides_new) == 3

    peptide_list = ['PEPTIDE']
    mods_variable = ['pP','pS']
    isoforms_max = 1024

    peptides_new = add_variable_mods(peptide_list, mods_variable, isoforms_max, None)

    assert len(peptides_new) == 4

test_add_variable_mods()_____no_output_____

### Terminal Modifications - Fixed

To handle terminal modifications, we use the following convention:

* `<` for the left side (N-terminal)
* `>` for the right side (C-Terminal)

Additionally, if we want to have a terminal modification on any AA we indicate this `^`._____no_output_____

#export
def add_fixed_mod_terminal(peptides:list, mod:str)->list:
    """
    Adds fixed terminal modifications
    Args:
        peptides (list of str): peptide list.
        mod (str): n-term mod contains '<^' (e.g. a<^ for Acetyl@N-term); c-term mod contains '>^'.
    Raises:
        "Invalid fixed terminal modification 'mod name'" for the given mod.
    Returns:
        list (of str): list of peptides with modification added.
    """
    # < for left side (N-Term), > for right side (C-Term)
    if "<^" in mod: #Any n-term, e.g. a<^
        peptides = [mod[:-2] + peptide for peptide in peptides]
    elif ">^" in mod: #Any c-term, e.g. a>^
        peptides = [peptide[:-1] + mod[:-2] + peptide[-1] for peptide in peptides]
    elif "<" in mod: #only if specific AA, e.g. ox<C
        peptides = [peptide[0].replace(mod[-1], mod[:-2]+mod[-1]) + peptide[1:] for peptide in peptides]
    elif ">" in mod:
        peptides = [peptide[:-1] + peptide[-1].replace(mod[-1], mod[:-2]+mod[-1]) for peptide in peptides]
    else:
        # This should not happen
        raise ("Invalid fixed terminal modification {}.".format(mod))
    return peptides

def add_fixed_mods_terminal(peptides:list, mods_fixed_terminal:list, **kwargs)->list:
    """
    Wrapper to add fixed mods on sequences and lists of mods
    Args:
        peptides (list of str): peptide list.
        mods_fixed_terminal (list of str): list of fixed terminal mods.
    Raises:
        "Invalid fixed terminal modification {mod}" exception for the given mod.
    Returns:
        list (of str): list of peptides with modification added.
    """
    if mods_fixed_terminal == []:
        return peptides
    else:
        # < for left side (N-Term), > for right side (C-Term)
        for key in mods_fixed_terminal:
            peptides = add_fixed_mod_terminal(peptides, key)
        return peptides_____no_output_____peptide = ['AMAMA']

print(f'Starting with peptide {peptide}')
print('Any n-term modified with x (x<^):', add_fixed_mods_terminal(peptide, ['x<^']))
print('Any c-term modified with x (x>^):', add_fixed_mods_terminal(peptide, ['x>^']))
print('Only A on n-term modified with x (x<A):', add_fixed_mods_terminal(peptide, ['x<A']))
print('Only A on c-term modified with x (xA']))Starting with peptide ['AMAMA']
Any n-term modified with x (x<^): ['xAMAMA']
Any c-term modified with x (x>^): ['AMAMxA']
Only A on n-term modified with x (x<A): ['xAMAMA']
Only A on c-term modified with x (x<A): ['AMAMxA']
#hide

def test_add_fixed_mods_terminal():
    peptide = ['AMAMA']

    peptides_new = add_fixed_mods_terminal(peptide, [])
    assert peptides_new == peptide

    #Any N-term
    peptides_new = add_fixed_mods_terminal(peptide, ['x<^'])
    assert peptides_new == ['xAMAMA']

    #Any C-term
    peptides_new = add_fixed_mods_terminal(peptide, ['x>^'])
    assert peptides_new == ['AMAMxA']

    #Selected N-term
    peptides_new = add_fixed_mods_terminal(peptide, ['x<A'])
    assert peptides_new == ['xAMAMA']

    peptides_new = add_fixed_mods_terminal(peptide, ['x<C'])
    assert peptides_new == peptide

    #Selected C-term
    peptides_new = add_fixed_mods_terminal(peptide, ['x>A'])
    assert peptides_new == ['AMAMxA']

    peptides_new = add_fixed_mods_terminal(peptide, ['x>C'])
    assert peptides_new == peptide

test_add_fixed_mods_terminal()_____no_output_____

### Terminal Modifications - Variable

Lastly, to handle terminal variable modifications, we use the function `add_variable_mods_terminal`. As the modification can only be at the terminal end, this function only adds a peptide where the terminal end is modified._____no_output_____

#export
def add_variable_mods_terminal(peptides:list, mods_variable_terminal:list, **kwargs)->list:
    """
    Function to add variable terminal modifications.
    Args:
        peptides (list of str): peptide list.
        mods_variable_terminal (list of str): list of variable terminal mods.
    Returns:
        list (of str): list of peptides with modification added.
    """
    if not mods_variable_terminal:
        return peptides
    else:
        new_peptides_n = peptides.copy()

        for key in mods_variable_terminal:
            if "<" in key:
                # Only allow one variable mod on one end
                new_peptides_n.extend(
                    add_fixed_mod_terminal(peptides, key)
                )
        new_peptides_n = get_unique_peptides(new_peptides_n)
        # N complete, let's go for c-terminal
        new_peptides_c = new_peptides_n
        for key in mods_variable_terminal:
            if ">" in key:
                # Only allow one variable mod on one end
                new_peptides_c.extend(
                    add_fixed_mod_terminal(new_peptides_n, key)
                )

        return get_unique_peptides(new_peptides_c)

def get_unique_peptides(peptides:list) -> list:
    """
    Function to return unique elements from list.
    Args:
        peptides (list of str): peptide list.
    Returns:
        list (of str): list of peptides (unique).
    """
    return list(set(peptides))_____no_output_____peptide_list = ['AMAMA']
add_variable_mods_terminal(peptide_list, ['x<^'])_____no_output_____#hide
def test_add_variable_mods_terminal():
    peptide_list = ['AMAMA']

    peptides_new = add_variable_mods_terminal(peptide_list, [])
    assert peptides_new == peptide

    #Any N-term
    peptides_new = add_variable_mods_terminal(peptide_list, ['x<^'])
    assert set(peptides_new) == set(['xAMAMA', 'AMAMA'])

test_add_variable_mods_terminal()_____no_output_____

### Generating Peptides

Lastly, we put all the functions into a wrapper `generate_peptides`. It will accept a peptide and a dictionary with settings so that we can get all modified peptides._____no_output_____

#export
def generate_peptides(peptide:str, **kwargs)->list:
    """
    Wrapper to get modified peptides (fixed and variable mods) from a peptide.

    Args:
        peptide (str): the given peptide sequence.
    Returns:
        list (of str): all modified peptides.

    TODO:
        There can be some edge-cases which are not defined yet.
        Example:
        Setting the same fixed modification - once for all peptides and once for only terminal for the protein.
        The modification will then be applied twice.
    """
    mod_peptide = add_fixed_mods_terminal([peptide], kwargs['mods_fixed_terminal_prot'])
    mod_peptide = add_variable_mods_terminal(mod_peptide, kwargs['mods_variable_terminal_prot'])

    peptides = []
    [peptides.extend(cleave_sequence(_, **kwargs)) for _ in mod_peptide]

    peptides = [_ for _ in peptides if check_peptide(_, constants.AAs)]

    isoforms_max = kwargs['isoforms_max']

    all_peptides = []
    for peptide in peptides: #1 per, limit the number of isoforms

        #Regular peptides
        mod_peptides = add_fixed_mods([peptide], **kwargs)
        mod_peptides = add_fixed_mods_terminal(mod_peptides, **kwargs)
        mod_peptides = add_variable_mods_terminal(mod_peptides, **kwargs)

        kwargs['isoforms_max'] = isoforms_max - len(mod_peptides)
        mod_peptides = add_variable_mods(mod_peptides, **kwargs)

        all_peptides.extend(mod_peptides)

        #Decoys:
        decoy_peptides = get_decoys([peptide], **kwargs)

        mod_peptides_decoy = add_fixed_mods(decoy_peptides, **kwargs)
        mod_peptides_decoy = add_fixed_mods_terminal(mod_peptides_decoy, **kwargs)
        mod_peptides_decoy = add_variable_mods_terminal(mod_peptides_decoy, **kwargs)

        kwargs['isoforms_max'] = isoforms_max - len(mod_peptides_decoy)

        mod_peptides_decoy = add_variable_mods(mod_peptides_decoy, **kwargs)

        mod_peptides_decoy = add_decoy_tag(mod_peptides_decoy)

        all_peptides.extend(mod_peptides_decoy)

    return all_peptides

def check_peptide(peptide:str, AAs:set)->bool:
    """
    Check if the peptide contains non-AA letters.
    Args:
        peptide (str): peptide sequence.
        AAs (set): the set of legal amino acids. See alphapept.constants.AAs
    Returns:
        bool: True if all letters in the peptide is the subset of AAs, otherwise False
    """
    if set([_ for _ in peptide if _.isupper()]).issubset(AAs):
        return True
    else:
        return False_____no_output_____kwargs = {}

kwargs["protease"] = "trypsin"
kwargs["n_missed_cleavages"] = 2
kwargs["pep_length_min"] = 6
kwargs["pep_length_max"] = 27
kwargs["mods_variable"] = ["oxM"]
kwargs["mods_variable_terminal"] = []
kwargs["mods_fixed"] = ["cC"]
kwargs["mods_fixed_terminal"] = []
kwargs["mods_fixed_terminal_prot"] = []
kwargs["mods_variable_terminal_prot"]  = ['a<^']
kwargs["isoforms_max"] = 1024
kwargs["n_modifications_max"] = None

generate_peptides('PEPTIDEM', **kwargs)_____no_output_____#hide
def test_generate_peptides():
    kwargs = {}

    kwargs["protease"] = "trypsin"
    kwargs["n_missed_cleavages"] = 2
    kwargs["pep_length_min"] = 6
    kwargs["pep_length_max"] = 27
    kwargs["mods_variable"] = ["oxM"]
    kwargs["mods_variable_terminal"] = []
    kwargs["mods_fixed"] = ["cC"]
    kwargs["mods_fixed_terminal"] = []
    kwargs["mods_fixed_terminal_prot"] = []
    kwargs["mods_variable_terminal_prot"]  = []
    kwargs["isoforms_max"] = 1024
    kwargs['pseudo_reverse'] = True
    kwargs["n_modifications_max"] = None

    peps = generate_peptides('PEPTIDEM', **kwargs)
    assert set(peps) == set(['PEPTIDEM', 'PEPTIDEoxM', 'EDITPEPM_decoy', 'EDITPEPoxM_decoy'])

test_generate_peptides()_____no_output_____

## Mass Calculations

Using the `mass_dict` from `constants` and being able to parse sequences with `parse`, one can simply look up the masses for each modified or unmodified amino acid and add everything up.

### Precursor

To calculate the mass of the neutral precursor, we start with the mass of an $H_2O$ and add the masses of all amino acids of the sequence._____no_output_____

#export
from numba import njit
from numba.typed import List
import numpy as np
import numba

@njit
def get_precmass(parsed_pep:list, mass_dict:numba.typed.Dict)->float:
    """
    Calculate the mass of the neutral precursor
    Args:
        parsed_pep (list or numba.typed.List of str): the list of amino acids and modified amono acids.
        mass_dict (numba.typed.Dict): key is the amino acid or the modified amino acid, and the value is the mass.
    Returns:
        float: the peptide neutral mass.
    """
    tmass = mass_dict["H2O"]
    for _ in parsed_pep:
        tmass += mass_dict[_]

    return tmass_____no_output_____get_precmass(parse('PEPTIDE'), constants.mass_dict)_____no_output_____#hide
def test_get_precmass():

    precmass = get_precmass(parse('PEPTIDE'), constants.mass_dict)

    assert np.allclose(precmass, 799.3599642034599)

test_get_precmass()_____no_output_____

### Fragments

Likewise, we can calculate the masses of the fragment ions. We employ two functions: `get_fragmass` and `get_frag_dict`.

`get_fragmass` is a fast, `numba`-compatible function that calculates the fragment masses and returns an array indicating whether the ion-type was `b` or `y`.

`get_frag_dict` instead is not `numba`-compatible and hence a bit slower. It returns a dictionary with the respective ion and can be used for plotting theoretical spectra._____no_output_____

#export
import numba

@njit
def get_fragmass(parsed_pep:list, mass_dict:numba.typed.Dict)->tuple:
    """
    Calculate the masses of the fragment ions
    Args:
        parsed_pep (numba.typed.List of str): the list of amino acids and modified amono acids.
        mass_dict (numba.typed.Dict): key is the amino acid or the modified amino acid, and the value is the mass.
    Returns:
        Tuple[np.ndarray(np.float64), np.ndarray(np.int8)]: the fragment masses and the fragment types (represented as np.int8).
        For a fragment type, positive value means the b-ion, the value indicates the position (b1, b2, b3...); the negative value means
        the y-ion, the absolute value indicates the position (y1, y2, ...).
    """
    n_frags = (len(parsed_pep) - 1) * 2

    frag_masses = np.zeros(n_frags, dtype=np.float64)
    frag_type = np.zeros(n_frags, dtype=np.int8)

    # b-ions > 0
    n_frag = 0

    frag_m = mass_dict["Proton"]
    for idx, _ in enumerate(parsed_pep[:-1]):
        frag_m += mass_dict[_]
        frag_masses[n_frag] = frag_m
        frag_type[n_frag] = (idx+1)
        n_frag += 1

    # y-ions < 0
    frag_m = mass_dict["Proton"] + mass_dict["H2O"]
    for idx, _ in enumerate(parsed_pep[::-1][:-1]):
        frag_m += mass_dict[_]
        frag_masses[n_frag] = frag_m
        frag_type[n_frag] = -(idx+1)
        n_frag += 1

    return frag_masses, frag_type_____no_output_____get_fragmass(parse('PEPTIDE'), constants.mass_dict)_____no_output_____#hide
def test_get_fragmass():

    frag_masses, frag_type = get_fragmass(parse('PEPTIDE'), constants.mass_dict)

    ref_masses = np.array([ 98.06004033, 227.10263343, 324.15539729, 425.20307579,
        538.28713979, 653.31408289, 148.06043425, 263.08737735,
        376.17144135, 477.21911985, 574.27188371, 703.31447681])

    assert np.allclose(frag_masses, ref_masses)

test_get_fragmass()_____no_output_____#export
def get_frag_dict(parsed_pep:list, mass_dict:dict)->dict:
    """
    Calculate the masses of the fragment ions
    Args:
        parsed_pep (list or numba.typed.List of str): the list of amino acids and modified amono acids.
        mass_dict (numba.typed.Dict): key is the amino acid or the modified amino acid, and the value is the mass.
    Returns:
        dict{str:float}: key is the fragment type (b1, b2, ..., y1, y2, ...), value is fragment mass.
    """
    frag_dict = {}
    frag_masses, frag_type = get_fragmass(parsed_pep, mass_dict)

    for idx, _ in enumerate(frag_masses):

        cnt = frag_type[idx]
        if cnt > 0:
            identifier = 'b'
        else:
            identifier = 'y'
            cnt = -cnt
        frag_dict[identifier+str(cnt)] = _

    return frag_dict_____no_output_____get_frag_dict(parse('PEPTIDE'), constants.mass_dict)_____no_output_____#hide
def test_get_frag_dict():

    refdict = {'b1': 98.06004032687,
 'b2': 227.10263342686997,
 'b3': 324.15539728686997,
 'y1': 120.06551965033,
 'y2': 217.11828351033,
 'y3': 346.16087661033}

    newdict = get_frag_dict(parse('PEPT'), constants.mass_dict)

    for key in newdict.keys():

        assert np.allclose(refdict[key], newdict[key])

test_get_frag_dict()_____no_output_____

This allows us also to generate the theoretical isotopes for a fragment:_____no_output_____

import matplotlib.pyplot as plt
%matplotlib inline
peptide = 'PEPTIDE'

frag_dict = get_frag_dict(parse(peptide), constants.mass_dict)

db_frag = list(frag_dict.values())
db_int = [100 for _ in db_frag]

plt.figure(figsize=(10,5))
plt.vlines(db_frag, 0, db_int, "k", label="DB", alpha=0.2)

for _ in frag_dict.keys():
    plt.text(frag_dict[_], 104, _, fontsize=12, alpha = 0.8)

plt.title('Theoretical Spectrum for {}'.format(peptide))
plt.xlabel('Mass')
plt.ylabel('Intensity')
plt.ylim([0,110])
plt.show()_____no_output_____

### Spectra

The function `get_spectrum` returns a tuple with the following content:

* precursor mass
* peptide sequence
* fragmasses
* fragtypes

Likewise, `get_spectra` returns a list of tuples. We employ a list of tuples here as this way, we can sort them easily by precursor mass._____no_output_____

#export
@njit
def get_spectrum(peptide:str, mass_dict:numba.typed.Dict)->tuple:
    """
    Get neutral peptide mass, fragment masses and fragment types for a peptide
    Args:
        peptide (str): the (modified) peptide.
        mass_dict (numba.typed.Dict): key is the amino acid or modified amino acid, and the value is the mass.
    Returns:
        Tuple[float, str, np.ndarray(np.float64), np.ndarray(np.int8)]: (peptide mass, peptide, fragment masses, fragment_types), for fragment types, see get_fragmass.
    """
    parsed_peptide = parse(peptide)

    fragmasses, fragtypes = get_fragmass(parsed_peptide, mass_dict)
    sortindex = np.argsort(fragmasses)
    fragmasses = fragmasses[sortindex]
    fragtypes = fragtypes[sortindex]

    precmass = get_precmass(parsed_peptide, mass_dict)

    return (precmass, peptide, fragmasses, fragtypes)

@njit
def get_spectra(peptides:numba.typed.List, mass_dict:numba.typed.Dict)->List:
    """
    Get neutral peptide mass, fragment masses and fragment types for a list of peptides
    Args:
        peptides (list of str): the (modified) peptide list.
        mass_dict (numba.typed.Dict): key is the amino acid or modified amino acid, and the value is the mass.
    Raises:
        Unknown exception and pass.
    Returns:
        list of Tuple[float, str, np.ndarray(np.float64), np.ndarray(np.int8)]: See get_spectrum.
    """
    spectra = List()

    for i in range(len(peptides)):
        try:
            spectra.append(get_spectrum(peptides[i], mass_dict))
        except Exception: #TODO: This is to fix edge cases when having multiple modifications on the same AA.
            pass

    return spectra_____no_output_____print(get_spectra(List(['PEPTIDE']), constants.mass_dict))[(799.35996420346, 'PEPTIDE', array([ 98.06004033, 148.06043425, 227.10263343, 263.08737735,
       324.15539729, 376.17144135, 425.20307579, 477.21911985,
       538.28713979, 574.27188371, 653.31408289, 703.31447681]), array([ 1, -1,  2, -2,  3, -3,  4, -4,  5, -5,  6, -6], dtype=int8))]
#hide
def test_get_spectra():

    spectra = get_spectra(List(['PEPTIDE']), constants.mass_dict)

    precmass, peptide, frags, fragtypes = spectra[0]

    assert np.allclose(precmass, 799.3599642034599)

    assert peptide == 'PEPTIDE'

    assert np.allclose(frags, np.array([ 98.06004033, 148.06043425, 227.10263343, 263.08737735,
       324.15539729, 376.17144135, 425.20307579, 477.21911985,
       538.28713979, 574.27188371, 653.31408289, 703.31447681]))

test_get_spectra()_____no_output_____

## Reading FASTA

To read FASTA files, we use the `SeqIO` module from the `Biopython` library. This is a generator expression so that we read one FASTA entry after another until the `StopIteration` is reached, which is implemented in `read_fasta_file`. Additionally, we define the function `read_fasta_file_entries` that simply counts the number of FASTA entries.

All FASTA entries that contain AAs which are not in the mass_dict can be checked with `check_sequence` and will be ignored._____no_output_____

#export
from Bio import SeqIO
import os
from glob import glob
import logging

def read_fasta_file(fasta_filename:str=""):
    """
    Read a FASTA file line by line
    Args:
        fasta_filename (str): fasta.
    Yields:
        dict {id:str, name:str, description:str, sequence:str}: protein information.
    """
    with open(fasta_filename, "rt") as handle:
        iterator = SeqIO.parse(handle, "fasta")
        while iterator:
            try:
                record = next(iterator)
                parts = record.id.split("|")  # pipe char
                if len(parts) > 1:
                    id = parts[1]
                else:
                    id = record.name
                sequence = str(record.seq)
                entry = {
                    "id": id,
                    "name": record.name,
                    "description": record.description,
                    "sequence": sequence,
                }

                yield entry
            except StopIteration:
                break

def read_fasta_file_entries(fasta_filename=""):
    """
    Function to count entries in fasta file
    Args:
        fasta_filename (str): fasta.
    Returns:
        int: number of entries.
    """
    with open(fasta_filename, "rt") as handle:
        iterator = SeqIO.parse(handle, "fasta")
        count = 0
        while iterator:
            try:
                record = next(iterator)
                count+=1
            except StopIteration:
                break

        return count

def check_sequence(element:dict, AAs:set, verbose:bool = False)->bool:
    """
    Checks wheter a sequence from a FASTA entry contains valid AAs
    Args:
        element (dict): fasta entry of the protein information.
        AAs (set): a set of amino acid letters.
        verbose (bool): logging the invalid amino acids.
    Returns:
        bool: False if the protein sequence contains non-AA letters, otherwise True.
    """
    if not set(element['sequence']).issubset(AAs):
        unknown = set(element['sequence']) - set(AAs)
        if verbose:
            logging.error(f'This FASTA entry contains unknown AAs {unknown} - Peptides with unknown AAs will be skipped: \n {element}\n')
        return False
    else:
        return True

_____no_output_____#load example fasta file

fasta_path = '../testfiles/test.fasta'

list(read_fasta_file(fasta_path))[0]_____no_output_____

## Peptide Dictionary

In order to efficiently store peptides, we rely on the Python dictionary. The idea is to have a dictionary with peptides as keys and indices to proteins as values. This way, one can quickly look up to which protein a peptide belongs to. The function `add_to_pept_dict` uses a regular python dictionary and allows to add peptides and stores indices to the originating proteins as a list. If a peptide is already present in the dictionary, the list is appended. The function returns a list of `added_peptides`, which were not present in the dictionary yet. One can use the function `merge_pept_dicts` to merge multiple peptide dicts._____no_output_____

#export
def add_to_pept_dict(pept_dict:dict, new_peptides:list, i:int)->tuple:
    """
    Add peptides to the peptide dictionary
    Args:
        pept_dict (dict): the key is peptide sequence, and the value is protein id list indicating where the peptide is from.
        new_peptides (list): the list of peptides to be added to pept_dict.
        i (int): the protein id where new_peptides are from.
    Returns:
        dict: same as the pept_dict in the arguments.
        list (of str): the peptides from new_peptides that not in the pept_dict.
    """
    added_peptides = List()
    for peptide in new_peptides:
        if peptide in pept_dict:
            if i not in pept_dict[peptide]:
                pept_dict[peptide].append(i)
        else:
            pept_dict[peptide] = [i]
            added_peptides.append(peptide)

    return pept_dict, added_peptides_____no_output_____pept_dict = {}
new_peptides = ['ABC','DEF']

pept_dict, added_peptides = add_to_pept_dict(pept_dict, new_peptides, 0)

new_peptides = ['DEF','GHI']

pept_dict, added_peptides = add_to_pept_dict(pept_dict, new_peptides, 1)

print(pept_dict){'ABC': [0], 'DEF': [0, 1], 'GHI': [1]}
#hide

def test_add_to_pept_dict():
    pept_dict = {}
    new_peptides = ['ABC','DEF']

    pept_dict, added_peptides = add_to_pept_dict(pept_dict, new_peptides, 0)

    new_peptides = ['DEF','GHI']

    pept_dict, added_peptides = add_to_pept_dict(pept_dict, new_peptides, 1)

    assert pept_dict == {'ABC': [0], 'DEF': [0, 1], 'GHI': [1]}

test_add_to_pept_dict()_____no_output_____#export

def merge_pept_dicts(list_of_pept_dicts:list)->dict:
    """
    Merge a list of peptide dict into a single dict.
    Args:
        list_of_pept_dicts (list of dict): the key of the pept_dict is peptide sequence, and the value is protein id list indicating where the peptide is from.
    Returns:
        dict: the key is peptide sequence, and the value is protein id list indicating where the peptide is from.
    """
    if len(list_of_pept_dicts) == 0:
        raise ValueError('Need to pass at least 1 element.')

    new_pept_dict = list_of_pept_dicts[0]

    for pept_dict in list_of_pept_dicts[1:]:

        for key in pept_dict.keys():
            if key in new_pept_dict:
                for element in pept_dict[key]:
                    new_pept_dict[key].append(element)
            else:
                new_pept_dict[key] = pept_dict[key]

    return new_pept_dict_____no_output_____pept_dict_1 = {'ABC': [0], 'DEF': [0, 1], 'GHI': [1]}
pept_dict_2 = {'ABC': [3,4], 'JKL': [5, 6], 'MNO': [7]}

merge_pept_dicts([pept_dict_1, pept_dict_2])_____no_output_____#hide
def test_merge_pept_dicts():
    pept_dict_1 = {'ABC': [0], 'DEF': [0, 1], 'GHI': [1]}
    pept_dict_2 = {'ABC': [3,4], 'JKL': [5, 6], 'MNO': [7]}

    assert merge_pept_dicts([pept_dict_1, pept_dict_2]) == {'ABC': [0, 3, 4], 'DEF': [0, 1], 'GHI': [1], 'JKL': [5, 6], 'MNO': [7]}

test_merge_pept_dicts()_____no_output_____

## Generating a database

To wrap everything up, we employ two functions, `generate_database` and `generate_spectra`. The first one reads a FASTA file and generates a list of peptides, as well as the peptide dictionary and an ordered FASTA dictionary to be able to look up the protein indices later. For the `callback` we first read the whole FASTA file to determine the total number of entries in the FASTA file.  For a typical FASTA file of 30 Mb with 40k entries, this should take less than a second. The progress of the digestion is monitored by processing the FASTA file one by one.
The function `generate_spectra` then calculates precursor masses and fragment ions. Here, we split the total_number of sequences in `1000` steps to be able to track progress with the `callback`._____no_output_____

#export
from collections import OrderedDict

def generate_fasta_list(fasta_paths:list, callback = None, **kwargs)->tuple:
    """
    Function to generate a database from a fasta file
    Args:
        fasta_paths (str or list of str): fasta path or a list of fasta paths.
        callback (function, optional): callback function.
    Returns:
        fasta_list (list of dict): list of protein entry dict {id:str, name:str, description:str, sequence:str}.
        fasta_dict (dict{int:dict}): the key is the protein id, the value is the protein entry dict.
    """
    fasta_list = []

    fasta_dict = OrderedDict()

    fasta_index = 0

    if type(fasta_paths) is str:
        fasta_paths = [fasta_paths]
        n_fastas = 1

    elif type(fasta_paths) is list:
        n_fastas = len(fasta_paths)

    for f_id, fasta_file in enumerate(fasta_paths):
        n_entries = read_fasta_file_entries(fasta_file)

        fasta_generator = read_fasta_file(fasta_file)

        for element in fasta_generator:
            check_sequence(element, constants.AAs)
            fasta_list.append(element)
            fasta_dict[fasta_index] = element
            fasta_index += 1

    return fasta_list, fasta_dict

_____no_output_____#hide

def test_generate_fasta_list():
    fasta_list, fasta_dict = generate_fasta_list('../testfiles/test.fasta')
    assert len(fasta_list) == 17
    assert fasta_dict[0]['name'] == 'sp|A0PJZ0|A20A5_HUMAN'

test_generate_fasta_list()_____no_output_____#export

def generate_database(mass_dict:dict, fasta_paths:list, callback = None, **kwargs)->tuple:
    """
    Function to generate a database from a fasta file
    Args:
        mass_dict (dict): not used, will be removed in the future.
        fasta_paths (str or list of str): fasta path or a list of fasta paths.
        callback (function, optional): callback function.
    Returns:
        to_add (list of str): non-redundant (modified) peptides to be added.
        pept_dict (dict{str:list of int}): the key is peptide sequence, and the value is protein id list indicating where the peptide is from.
        fasta_dict (dict{int:dict}): the key is the protein id, the value is the protein entry dict {id:str, name:str, description:str, sequence:str}.
    """
    to_add = List()
    fasta_dict = OrderedDict()
    fasta_index = 0

    pept_dict = {}

    if type(fasta_paths) is str:
        fasta_paths = [fasta_paths]
        n_fastas = 1

    elif type(fasta_paths) is list:
        n_fastas = len(fasta_paths)

    for f_id, fasta_file in enumerate(fasta_paths):
        n_entries = read_fasta_file_entries(fasta_file)

        fasta_generator = read_fasta_file(fasta_file)

        for element in fasta_generator:

            fasta_dict[fasta_index] = element
            mod_peptides = generate_peptides(element["sequence"], **kwargs)
            pept_dict, added_seqs = add_to_pept_dict(pept_dict, mod_peptides, fasta_index)
            if len(added_seqs) > 0:
                to_add.extend(added_seqs)

            fasta_index += 1

            if callback:
                callback(fasta_index/n_entries/n_fastas+f_id)

    return to_add, pept_dict, fasta_dict_____no_output_____#hide

def test_generate_database():

    from alphapept.constants import mass_dict
    from alphapept.settings import load_settings
    from alphapept.paths import DEFAULT_SETTINGS_PATH

    settings = load_settings(DEFAULT_SETTINGS_PATH)

    to_add, pept_dict, fasta_dict = generate_database(mass_dict, ['../testfiles/test.fasta'], **settings['fasta'])

    assert len(to_add) == 3240 #This will break if the deafult settings are changed
    assert 'GQTVLGSIDHLYTGSGYR' in pept_dict
    assert fasta_dict[0]['name'] == 'sp|A0PJZ0|A20A5_HUMAN'

test_generate_database()_____no_output_____#export

def generate_spectra(to_add:list, mass_dict:dict, callback = None)->list:
    """
    Function to generate spectra list database from a fasta file
    Args:
        to_add (list):
        mass_dict (dict{str:float}): amino acid mass dict.
        callback (function, optional): callback function. (Default: None)
    Returns:
        list (of tuple): list of (peptide mass, peptide, fragment masses, fragment_types), see get_fragmass.
    """

    if len(to_add) > 0:

        if callback: #Chunk the spectra to get a progress_bar
            spectra = []

            stepsize = int(np.ceil(len(to_add)/1000))

            for i in range(0, len(to_add), stepsize):
                sub = to_add[i:i + stepsize]
                spectra.extend(get_spectra(sub, mass_dict))
                callback((i+1)/len(to_add))

        else:
            spectra = get_spectra(to_add, mass_dict)
    else:
        raise ValueError("No spectra to generate.")

    return spectra_____no_output_____#hide

def test_generate_spectra():
    from alphapept.constants import mass_dict
    from numba.typed import List

    to_add = List(['PEPTIDE'])
    spectra = generate_spectra(to_add, mass_dict)
    assert np.allclose(spectra[0][0], 799.35996420346)
    assert spectra[0][1] == 'PEPTIDE'

test_generate_spectra()_____no_output_____

## Parallelized version

To speed up spectra generated, one can use the parallelized version. The function `generate_database_parallel` reads an entire FASTA file and splits it into multiple blocks. Each block will be processed, and the generated pept_dicts will be merged._____no_output_____

#export
from typing import Generator

def block_idx(len_list:int, block_size:int = 1000)->list:
    """
    Helper function to split length into blocks
    Args:
        len_list (int): list length.
        block_size (int, optional, default 1000): size per block.
    Returns:
        list[(int, int)]: list of (start, end) positions of blocks.
    """
    blocks = []
    start = 0
    end = 0

    while end <= len_list:
        end += block_size
        blocks.append((start, end))
        start = end

    return blocks

def blocks(l:int, n:int)->Generator[list, None, None]:
    """
    Helper function to create blocks from a given list
    Args:
        l (list): the list
        n (int): size per block
    Returns:
        Generator: List with splitted elements
    """
    n = max(1, n)
    return (l[i:i+n] for i in range(0, len(l), n))_____no_output_____#hide

def test_block_idx():
    assert block_idx(100, 50) == [(0, 50), (50, 100), (100, 150)]

test_block_idx()

def test_blocks():
    assert list(blocks([1,2,3,4], 2)) == [[1, 2], [3, 4]]

test_blocks()_____no_output_____#export

from multiprocessing import Pool
from alphapept import constants
mass_dict = constants.mass_dict

#This function is a wrapper function and to be tested by the integration test
def digest_fasta_block(to_process:tuple)-> (list, dict):
    """
    Digest and create spectra for a whole fasta_block for multiprocessing. See generate_database_parallel.
    """

    fasta_index, fasta_block, settings = to_process

    to_add = List()

    f_index = 0

    pept_dict = {}
    for element in fasta_block:
        sequence = element["sequence"]
        mod_peptides = generate_peptides(sequence, **settings['fasta'])
        pept_dict, added_peptides = add_to_pept_dict(pept_dict, mod_peptides, fasta_index+f_index)
        if len(added_peptides) > 0:
            to_add.extend(added_peptides)
        f_index += 1

    spectra = []
    if len(to_add) > 0:
        for specta_block in blocks(to_add, settings['fasta']['spectra_block']):
            spectra.extend(generate_spectra(specta_block, mass_dict))

    return (spectra, pept_dict)

import alphapept.performance

#This function is a wrapper function and to be tested by the integration test
def generate_database_parallel(settings:dict, callback = None):
    """
    Function to generate a database from a fasta file in parallel.
    Args:
        settings: alphapept settings.
    Returns:
        list: theoretical spectra. See generate_spectra()
        dict: peptide dict. See add_to_pept_dict()
        dict: fasta_dict. See generate_fasta_list()
    """

    n_processes = alphapept.performance.set_worker_count(
        worker_count=settings['general']['n_processes'],
        set_global=False
    )

    fasta_list, fasta_dict = generate_fasta_list(fasta_paths = settings['experiment']['fasta_paths'], **settings['fasta'])

    logging.info(f'FASTA contains {len(fasta_list):,} entries.')

    blocks = block_idx(len(fasta_list), settings['fasta']['fasta_block'])

    to_process = [(idx_start, fasta_list[idx_start:idx_end], settings) for idx_start, idx_end in  blocks]

    spectra = []
    pept_dicts = []
    with Pool(n_processes) as p:
        max_ = len(to_process)
        for i, _ in enumerate(p.imap_unordered(digest_fasta_block, to_process)):
            if callback:
                callback((i+1)/max_)
            spectra.extend(_[0])
            pept_dicts.append(_[1])

    spectra = sorted(spectra, key=lambda x: x[1])
    spectra_set = [spectra[idx] for idx in range(len(spectra)-1) if spectra[idx][1] != spectra[idx+1][1]]
    spectra_set.append(spectra[-1])

    pept_dict = merge_pept_dicts(pept_dicts)

    return spectra_set, pept_dict, fasta_dict_____no_output_____

### Parallel search on large files

In some cases (e.g., a lot of modifications or very large FASTA files), it will not be useful to save the database as it will consume too much memory. Here, we use the function `search_parallel` from search. It creates theoretical spectra on the fly and directly searches against them. As we cannot create a pept_dict here, we need to create one from the search results. For this, we group peptides by their FASTA index and generate a lookup dictionary that can be used as a pept_dict.

> Note that we are passing the settings argument here. Search results should be stored in the corresponding path in the `*.hdf` file._____no_output_____

#export
#This function is a wrapper function and to be tested by the integration test
def pept_dict_from_search(settings:dict):
    """
    Generates a peptide dict from a large search.
    """

    paths = settings['experiment']['file_paths']
    bases = [os.path.splitext(_)[0]+'.ms_data.hdf' for _ in paths]

    all_dfs = []
    for _ in bases:
        try:
            df = alphapept.io.MS_Data_File(_).read(dataset_name="peptide_fdr")
        except KeyError:
            df = pd.DataFrame()

        if len(df) > 0:
            all_dfs.append(df)

    if sum([len(_) for _ in all_dfs]) == 0:
        raise ValueError("No sequences present to concatenate.")

    df = pd.concat(all_dfs)

    df['fasta_index'] = df['fasta_index'].str.split(',')

    lst_col = 'fasta_index'

    df_ = pd.DataFrame({
          col:np.repeat(df[col].values, df[lst_col].str.len())
          for col in df.columns.drop(lst_col)}
        ).assign(**{lst_col:np.concatenate(df[lst_col].values)})[df.columns]

    df_['fasta_index'] = df_['fasta_index'].astype('int')
    df_grouped = df_.groupby(['sequence'])['fasta_index'].unique()

    pept_dict = {}
    for keys, vals in zip(df_grouped.index, df_grouped.values):
        pept_dict[keys] = vals.tolist()

    return pept_dict_____no_output_____

## Saving

To save the generated spectra, we rely on the HDF format. For this, we create a dictionary and save all the generated elements. The container will contain the following elements:

* `precursors`: An array containing the precursor masses
* `seqs`: An array containing the peptide sequences for the precursor masses
* `pept_dict`: A peptide dictionary to look up the peptides and return their FASTA index
* `fasta_dict`: A FASTA dictionary to look up the FASTA entry based on a pept_dict index
* `fragmasses`: An array containing the fragment masses. Unoccupied cells are filled with -1
* `fragtypes:`: An array containing the fragment types. 0 equals b-ions, and 1 equals y-ions. Unoccupied cells are filled with -1
* `bounds`: An integer array containing the upper bounds for the fragment masses/types array. This is needed to quickly slice the data.

All arrays are sorted according to the precursor mass.

> Note: To access the dictionaries such as `pept_dict` or `fasta_dict`, one needs to extract them using the `.item()` method like so: `container["pept_dict"].item()`._____no_output_____

#export
import alphapept.io
import pandas as pd

def save_database(spectra:list, pept_dict:dict, fasta_dict:dict, database_path:str, **kwargs):
    """
    Function to save a database to the *.hdf format. Write the database into hdf.

    Args:
        spectra (list): list: theoretical spectra. See generate_spectra().
        pept_dict (dict): peptide dict. See add_to_pept_dict().
        fasta_dict (dict): fasta_dict. See generate_fasta_list().
        database_path (str): Path to database.
    """

    precmasses, seqs, fragmasses, fragtypes = zip(*spectra)
    sortindex = np.argsort(precmasses)
    fragmasses = np.array(fragmasses, dtype=object)[sortindex]
    fragtypes = np.array(fragtypes, dtype=object)[sortindex]

    lens = [len(_) for _ in fragmasses]

    n_frags = sum(lens)

    frags = np.zeros(n_frags, dtype=fragmasses[0].dtype)
    frag_types = np.zeros(n_frags, dtype=fragtypes[0].dtype)

    indices = np.zeros(len(lens) + 1, np.int64)
    indices[1:] = lens
    indices = np.cumsum(indices)

    #Fill data

    for _ in range(len(indices)-1):

        start = indices[_]
        end = indices[_+1]
        frags[start:end] = fragmasses[_]
        frag_types[start:end] = fragtypes[_]

    to_save = {}

    to_save["precursors"] = np.array(precmasses)[sortindex]
    to_save["seqs"] = np.array(seqs, dtype=object)[sortindex]
    to_save["proteins"] = pd.DataFrame(fasta_dict).T

    to_save["fragmasses"] = frags
    to_save["fragtypes"] = frag_types
    to_save["indices"] = indices

    db_file = alphapept.io.HDF_File(database_path, is_new_file=True)
    for key, value in to_save.items():
        db_file.write(value, dataset_name=key)

    peps = np.array(list(pept_dict), dtype=object)
    indices = np.empty(len(peps) + 1, dtype=np.int64)
    indices[0] = 0
    indices[1:] = np.cumsum([len(pept_dict[i]) for i in peps])
    proteins = np.concatenate([pept_dict[i] for i in peps])

    db_file.write("peptides")
    db_file.write(
        peps,
        dataset_name="sequences",
        group_name="peptides"
    )
    db_file.write(
        indices,
        dataset_name="protein_indptr",
        group_name="peptides"
    )
    db_file.write(
        proteins,
        dataset_name="protein_indices",
        group_name="peptides"
    )_____no_output_____#export
import collections

def read_database(database_path:str, array_name:str=None)->dict:
    """
    Read database from hdf file.
    Args:
        database_path (str): hdf database file generate by alphapept.
        array_name (str): the dataset name to read
    return:
        dict: key is the dataset_name in hdf file, value is the python object read from the dataset_name
    """
    db_file = alphapept.io.HDF_File(database_path)
    if array_name is None:
        db_data = {
            key: db_file.read(
                dataset_name=key
            ) for key in db_file.read() if key not in (
                "proteins",
                "peptides"
            )
        }
        db_data["fasta_dict"] = np.array(
            collections.OrderedDict(db_file.read(dataset_name="proteins").T)
        )
        peps = db_file.read(dataset_name="sequences", group_name="peptides")
        protein_indptr = db_file.read(
            dataset_name="protein_indptr",
            group_name="peptides"
        )
        protein_indices = db_file.read(
            dataset_name="protein_indices",
            group_name="peptides"
        )
        db_data["pept_dict"] = np.array(
            {
                pep: (protein_indices[s: e]).tolist() for pep, s, e in zip(
                    peps,
                    protein_indptr[:-1],
                    protein_indptr[1:],
                )
            }
        )
        db_data["seqs"] = db_data["seqs"].astype(str)
    else:
        db_data = db_file.read(dataset_name=array_name)
    return db_data_____no_output_____#hide

def test_database_io():
    from alphapept.constants import mass_dict
    from numba.typed import List

    to_add = List(['PEPTIDE'])
    spectra = generate_spectra(to_add, mass_dict)

    fasta_list, fasta_dict = generate_fasta_list('../testfiles/test.fasta')

    database_path = '../testfiles/testdb.hdf'

    save_database(spectra, pept_dict, fasta_dict, database_path)

    assert list(read_database(database_path, 'seqs')) == ['PEPTIDE']
    assert np.allclose(list(read_database(database_path, 'precursors'))[0], spectra[0][0])

test_database_io()_____no_output_____#hide
from nbdev.export import *
notebook2script()Converted 00_settings.ipynb.
Converted 01_chem.ipynb.
Converted 02_io.ipynb.
Converted 03_fasta.ipynb.
Converted 04_feature_finding.ipynb.
Converted 05_search.ipynb.
Converted 06_score.ipynb.
Converted 07_recalibration.ipynb.
Converted 08_quantification.ipynb.
Converted 09_matching.ipynb.
Converted 10_constants.ipynb.
Converted 11_interface.ipynb.
Converted 12_performance.ipynb.
Converted 13_export.ipynb.
Converted additional_code.ipynb.
Converted contributing.ipynb.
Converted file_formats.ipynb.
Converted index.ipynb.
Converted Untitled.ipynb.