Software:NSynth

From HandWiki
Short description: Machine learning audio synthesizer
NSynth: Neural Audio Synthesis
Original author(s)Google Brain, Deep Mind, Magenta
Initial release6 April 2017; 7 years ago (2017-04-06)
Repositorygithub.com/magenta/magenta/tree/main/magenta/models/nsynth
Written inPython
TypeSoftware synthesizer
LicenseApache 2.0
Websitemagenta.tensorflow.org/nsynth

NSynth (a portmanteau of "Neural Synthesis") is a WaveNet-based autoencoder for synthesizing audio, outlined in a paper in April 2017.[1]

Overview

The model generates sounds through a neural network based synthesis, employing a WaveNet-style autoencoder to learn its own temporal embeddings from four different sounds.[2][3] Google then released an open source hardware interface for the algorithm called NSynth Super,[4] used by notable musicians such as Grimes and YACHT to generate experimental music using artificial intelligence.[5][6] The research and development of the algorithm was part of a collaboration between Google Brain, Magenta and DeepMind.[7]

Technology

Dataset

The NSynth dataset is composed of 305,979 one-shot instrumental notes featuring a unique pitch, timbre, and envelope, sampled from 1,006 instruments from commercial sample libraries.[8] For each instrument the dataset contains four-second 16 kHz audio snippets by ranging over every pitch of a standard MIDI piano, as well as five different velocities.[9] The dataset is made available under a Creative Commons Attribution 4.0 International (CC BY 4.0) license.[10]

Machine learning model

A spectral autoencoder model and a WaveNet autoencoder model are publicly available on GitHub.[11] The baseline model uses a spectrogram with fft_size 1024 and hop_size 256, MSE loss on the magnitudes, and the Griffin-Lim algorithm for reconstruction. The WaveNet model trains on mu-law encoded waveform chunks of size 6144. It learns embeddings with 16 dimensions that are downsampled by 512 in time.[12]

NSynth Super

NSynth Super
The NSynth Super front panel: a metal box with a bright colored screen input.
NSynth Super Front Panel
ManufacturerGoogle Brain, Google Creative Lab
Dates2018
Technical specifications
Synthesis typeNeural Network Sample-based synthesis
Input/output
Left-hand controlPitch bend, ADSR
External controlMIDI

In 2018 Google released a hardware interface for the NSynth algorithm, called NSynth Super, designed to provide an accessible physical interface to the algorithm for musicians to use in their artistic production.[13][14]

Design files, source code and internal components are released under an open source Apache License 2.0,[15] enabling hobbyists and musicians to freely build and use the instrument.[16] At the core of the NSynth Super there is a Raspberry Pi, extended with a custom printed circuit board to accommodate the interface elements.[17]

Influence

Despite not being publicly available as a commercial product, NSynth Super has been used by notable artists, including Grimes and YACHT.[18][19]

Grimes reported using the instrument in her 2020 studio album Miss Anthropocene.[5]

YACHT announced an extensive use of NSynth Super in their album Chain Tripping.[20]

Claire L. Evans compared the potential influence of the instrument to the Roland TR-808.[21]

The NSynth Super design was honored with a D&AD Yellow Pencil award in 2018.[22]

References

  1. ↑ Engel, Jesse; Resnick, Cinjon; Roberts, Adam; Dieleman, Sander; Eck, Douglas; Simonyan, Karen; Norouzi, Mohammad (2017). "Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders". arXiv:1704.01279 [cs.LG].
  2. ↑ Engel, Jesse; Resnick, Cinjon; Roberts, Adam; Dieleman, Sander; Eck, Douglas; Simonyan, Karen; Norouzi, Mohammad (2017). Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders. https://research.google/pubs/pub46119/. 
  3. ↑ Aaron van den Oord; Dieleman, Sander; Zen, Heiga; Simonyan, Karen; Vinyals, Oriol; Graves, Alex; Kalchbrenner, Nal; Senior, Andrew; Kavukcuoglu, Koray (2016). "WaveNet: A Generative Model for Raw Audio". arXiv:1609.03499 [cs.SD].
  4. ↑ "Google's open-source neural synth is creating totally new sounds". Wired UK. https://www.wired.co.uk/article/google-ai-nsynth-algorithm-music-creativity. 
  5. ↑ 5.0 5.1 "73 | Grimes (c) on Music, Creativity, and Digital Personae – Sean Carroll". https://www.preposterousuniverse.com/podcast/2019/11/18/73-grimes-c-on-music-creativity-and-digital-personae/. 
  6. ↑ Mattise, Nathan (2019-08-31). "How YACHT fed their old music to the machine and got a killer new album" (in en-us). https://arstechnica.com/gaming/2019/08/yachts-chain-tripping-is-a-new-landmark-for-ai-music-an-album-that-doesnt-suck/. 
  7. ↑ "NSynth: Neural Audio Synthesis". 6 April 2017. https://magenta.tensorflow.org/nsynth. 
  8. ↑ "NSynth Dataset" (in en-US). https://datasets.activeloop.ai/docs/ml/datasets/nsynth-dataset/. 
  9. ↑ Ramires, António; Serra, Xavier (2019). "Data Augmentation for Instrument Classification Robust to Audio Effects". arXiv:1907.08520 [cs.SD].
  10. ↑ "The NSynth Dataset". 5 April 2017. https://magenta.tensorflow.org/datasets/nsynth. 
  11. ↑ "NSynth: Neural Audio Synthesis". https://github.com/magenta/magenta/tree/main/magenta/models/nsynth. 
  12. ↑ Engel, Jesse; Resnick, Cinjon; Roberts, Adam; Dieleman, Sander; Eck, Douglas; Simonyan, Karen; Norouzi, Mohammad (2017). "Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders". arXiv:1704.01279 [cs.LG].
  13. ↑ "NSynth Super is an AI-backed touchscreen synth". 13 March 2018. https://www.theverge.com/circuitbreaker/2018/3/13/17114760/google-nsynth-super-ai-touchscreen-synth. 
  14. ↑ "Google built a musical instrument that uses AI and released the plans so you can make your own". 13 March 2018. https://www.cnbc.com/2018/03/13/google-launches-nsynth-super-ai-hardware-prototype.html. 
  15. ↑ "googlecreativelab/open-nsynth-super". April 1, 2021. https://github.com/googlecreativelab/open-nsynth-super. 
  16. ↑ "Open NSynth Super" (in en). https://hackaday.io/project/89396-open-nsynth-super. 
  17. ↑ "NSYNTH SUPER Hardware". https://github.com/googlecreativelab/open-nsynth-super/tree/master/pcb_hardware. 
  18. ↑ Mattise, Nathan. "How YACHT Used Machine Learning to Create Their New Album" (in en-US). Wired. ISSN 1059-1028. https://arstechnica.com/gaming/2019/08/yachts-chain-tripping-is-a-new-landmark-for-ai-music-an-album-that-doesnt-suck/. Retrieved 2023-01-19. 
  19. ↑ "Cover Story: Grimes is ready to play the villain". https://crackmagazine.net/article/long-reads/grimes-is-ready-to-play-the-villain/. 
  20. ↑ "What Machine-Learning Taught the Band YACHT About Themselves" (in en-US). 2019-09-18. https://losangeleno.com/people/what-machine-learning-taught-the-band-yacht-about-themselves/. 
  21. ↑ (in en) Music and Machine Learning (Google I/O'19), https://www.youtube.com/watch?v=pM9u9xcM_cs, retrieved 2023-01-19 
  22. ↑ "NSynth Super | Google Creative Lab | Google | D&AD Awards 2018 Pencil Winner | Interactive Design for Products | D&AD". https://www.dandad.org/awards/professional/2018/product-design/27149/nsynth-super/. 

Further reading

  • Engel, Jesse; Resnick, Cinjon; Roberts, Adam; Dieleman, Sander; Eck, Douglas; Simonyan, Karen; Norouzi, Mohammad (2017). "Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders". arXiv:1704.01279 [cs.LG].

External links




Retrieved from "https://handwiki.org/wiki/index.php?title=Software:NSynth&oldid=3467812"