You said open source. Open source is a type of licensure.

The entire point of licensure is legal pedantry.

And as far as your metaphor is concerned, pre-trained models are closer to pre-compiled binaries, which are expressly not considered Open Source according to the OSD.

From the approach section:

A Transformer sequence-to-sequence model is trained on various speech processing tasks, including multilingual speech recognition, speech translation, spoken language identification, and voice activity detection. These tasks are jointly represented as a sequence of tokens to be predicted by the decoder, allowing a single model to replace many stages of a traditional speech-processing pipeline. The multitask training format uses a set of special tokens that serve as task specifiers or classification targets.

This is not sufficient data information to recreate the model.

From the training data section:

The models are trained on 680,000 hours of audio and the corresponding transcripts collected from the internet. 65% of this data (or 438,000 hours) represents English-language audio and matched English transcripts, roughly 18% (or 126,000 hours) represents non-English audio and English transcripts, while the final 17% (or 117,000 hours) represents non-English audio and the corresponding transcript. This non-English data represents 98 different languages. As discussed in the accompanying paper, we see that performance on transcription in a given language is directly correlated with the amount of training data we employ in that language.

This is also insufficient data information and links to the paper itself for that data information.

Additionally, model cards =/= data cards. It’s an important distinction in AI training.

There are guides on how to Finetune the model yourself: https://huggingface.co/blog/fine-tune-whisper

Fine-tuning is not re-creating the model. This is an important distinction.

The OSAID has a pretty simple checklist for the OSAID definition: https://opensource.org/deepdive/drafts/the-open-source-ai-definition-checklist-draft-v-0-0-9

To go through the list of materials required to fit the OSAID:

Datasets Available under OSD-compliant license

Whisper does not provide the datasets.

Research paper Available under OSD-compliant license

The research paper is available, but does not fit an OSD-compliant license.

Technical report Available under OSD-compliant license

Whisper does not provide the technical report.

Data card Available under OSD-compliant license

Whisper provides the model card, but not the data card.

Oh and for the OSAID part, the only issue stopping Whisper from being considered open source as per the OSAID is that the information on the training data is published through arxiv, so using the data as written could present licensing issues.

The problem with just shipping AI model weights is that they run up against the issue of point 2 of the OSD:

The program must include source code, and must allow distribution in source code as well as compiled form. Where some form of a product is not distributed with source code, there must be a well-publicized means of obtaining the source code for no more than a reasonable reproduction cost, preferably downloading via the Internet without charge. The source code must be the preferred form in which a programmer would modify the program. Deliberately obfuscated source code is not allowed. Intermediate forms such as the output of a preprocessor or translator are not allowed.

AI models can’t be distributed purely as source because they are pre-trained. It’s the same as distributing pre-compiled binaries.

It’s the entire reason the OSAID exists:

1. The OSD doesn’t fit because it requires you distribute the source code in a non-preprocessed manner.
2. AIs can’t necessarily distribute the training data alongside the code that trains the model, so in order to help bridge the gap the OSI made the OSAID - as long as you fully document the way you trained the model so that somebody that has access to the training data you used can make a mostly similar set of weights, you fall within the OSAID

Edit: also the information about the training data has to be published in an OSD-equivalent license (such as creative Commons) so that using it doesn’t cause licensing issues with research paper print companies (like arxiv)

Whisper’s code and model weights are released under the MIT License. See LICENSE for further details. So that definitely meets the Open Source Definition on your first link.

Model weights by themselves do not qualify as “open source”, as the OSAID qualifies. Weights are not source.

Additional WER/CER metrics corresponding to the other models and datasets can be found in Appendix D.1, D.2, and D.4 of the paper, as well as the BLEU (Bilingual Evaluation Understudy) scores for translation in Appendix D.3.

This is not training data. These are testing metrics.

Edit: additionally, assuming you might have been talking about the link to the research paper. It’s not published under an OSD license. If it were this would qualify the model.

Those aren’t open source, neither by the OSI’s Open Source Definition nor by the OSI’s Open Source AI Definition.

The important part for the latter being a published listing of all the training data. (Trainers don’t have to provide the data, but they must provide at least a way to recreate the model given the same inputs).

Data information: Sufficiently detailed information about the data used to train the system, so that a skilled person can recreate a substantially equivalent system using the same or similar data. Data information shall be made available with licenses that comply with the Open Source Definition.

They are model-available if anything.

Those aren’t open source, neither by the OSI’s Open Source Definition nor by the OSI’s Open Source AI Definition.

The important part for the latter being a published listing of all the training data. (Trainers don’t have to provide the data, but they must provide at least a way to recreate the model given the same inputs).

Data information: Sufficiently detailed information about the data used to train the system, so that a skilled person can recreate a substantially equivalent system using the same or similar data. Data information shall be made available with licenses that comply with the Open Source Definition.

They are model-available if anything.

LLMs as they stand are already approaching the improvement flatline portion of the sigma curve due to marginal data requirements increasing exponentially.

It’s a known problem in the actual AI research field that nobody in private industry likes to talk about.

If it scores 40% this year it’ll marginally increase by 10% next year then 5% 3 years later and so on.

AI doesn’t follow Moore’s law.

You’re anthropomorphizing LLMs.

There’s a philosophical and neuroscuence concept called “Qualia,” which helps define the human experience. LLMs have no Qualia.

How do you do this with a tablet? Can you buy like a wheel sensor or something?

Unless something has changed recently you still have to submit builds to Nvidia to have them train the DLSS kernel for you, so FSR is substantially easier to integrate.

This is why I think eventually FSR will win over DLSS in the end, despite DLSS having better performance.

Not to rain on your parade but:

https://phys.org/news/2024-08-scientists-oceans-mars-deep.html

Using seismic activity to probe the interior of Mars, geophysicists have found evidence for a large underground reservoir of liquid water—enough to fill oceans on the planet’s surface.

It’s located in tiny cracks and pores in rock in the middle of the Martian crust, between 11.5 and 20 kilometers below the surface. Even on Earth, drilling a hole a kilometer deep is a challenge.

Ain’t nobody getting to that water anytime soon.

You can use nix alongside guix, it’ll just double-up the dependencies on disk:

services (append (list (service nix-service-type))
                    %base-services)))

Services are, in guix terms, any configuration change to a computer, so creating your own service 99% of the time is just extending etc-service-type and creating a variable interface to fill in the config file text yourself

Creating a service as in a daemon of some kind uses shepherd and involves extending shepherd-service-type or home-shepherd-service-type with your service description, depending on whether the service runs in root or user space.

Shepherd service configurations aren’t actually part of the guix spec(https://www.gnu.org/software/shepherd/manual/shepherd.html#Defining-Services), but still use Guile, so you can interoperate them super easily.

It’s important in guix to understand lisp pretty thoroughly, and knowing how to program lisp is still a very useful skill to have so I’d recommend learning it even if you never touch guix.

I use guix because, while it has a small community, the packaging language is one of the easiest I’ve ever used.

Every distro I’ve tried I’ve always run into having to wait on packages or support from someone else. The package transformation scheme like what nixos has is great but Nixlang sucks ass. Being able to do all that in lisp is much preferred.

Plus I like shepherd much more than any of the other process 0’s

I hope shepherd gets a mention in this series eventually

Facebook is trying to burn the forest around OpenAI and other closed models by removing the market for “models” by themselves, by releasing their own freely to the community. A lot of money is already pivoting away towards companies trying to find products that use the AI instead of the AI itself. Unless OpenAI pivots to something more substantial than just providing multimodal prompt completion they’re gonna find themselves without a lot of runway left.

I was never able to get appreciably better results from 11 labs than using some (minorly) trained RVC model :/ The long scripts problem is something pretty much any text-to-something model suffers from. The longer the context the lower the cohesion ends up.

I do rotoscoping with SDXL i2i and controlnet posing together. Without I found it tends to smear. Do you just do image2image?

Coqui for TTS, RVC UI for matching the TTS to the actor’s intonation, and DWPose -> controlnet applied to SDXL for rotoscoping

All the models I’ve used that do TTS/RVC and rotoscoping have definitely not produced professional results.