_________________________________________________________________________________
Hinton qualified that statement by adding, _"but I do think there's going to have to be quite a few conceptual breakthroughs."_ He explicitly conditioned his belief on future conceptual breakthroughs that have not been made yet. Also, his statement is not a prediction; it starts with "I do believe."
Here's the full quote, copied verbatim from the article:
_> I do believe deep learning is going to be able to do everything, but I do think there’s going to have to be quite a few conceptual breakthroughs. For example, in 2017 Ashish Vaswani et al. introduced transformers, which derive really good vectors representing word meanings. It was a conceptual breakthrough. It’s now used in almost all the very best natural-language processing. We’re going to need a bunch more breakthroughs like that._
Please don't criticize him or the article without first reading it in full.
Yeah, it's a bit tricky ... like saying "Turing Machines are going to be able to do everything, but we'll need quite a few conceptual breakthroughs." Hard to say what percent of progress we've made since 1936 toward artificial general intelligence (whatever that means).
In principle, yes, of course, because we use Turing machines for deep learning :-) but I think Hinton's point is much less broad than you imply: He believes "AGI" ultimately will be achieved with deep learning approaches, as opposed to other approaches, e.g., symbolic AI.
Yes, but "quite a few conceptual breakthroughs" leaves a lot of room open for if it would be at all similar to deep learning today.
Agree :-)
Hinton's comment was meant as, and is, an educated guess, expressed as a belief -- much like expressing belief that P≠NP, say, or that string theory will prove to be a dead end in Physics.
FWIW, I happen to agree with Hinton, but only time will tell if we're right!
_Now it’s hard to find anyone who disagrees, he says._
Either that's an accurate claim or representative of a cliqueyness and echo chamber approach to AGI in some quarters.
Either way it's depressing.
Does this mean we are at the top of another AI hype cycle? Anyone have a pool on when the next AI winter starts?
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
According to HN, AI winter has already started in 2017, 2018, 2019, 2020...
Winter will be gone forever, AI climate change it is :p
I’m tired of Hinton over promising. It’s not going to do everything and he’s promoting an irresponsible position.
Has there really been a huge breakthrough since the initial wave of CNNs? I guess transformers/attention, but I don’t consider GPT-3 to solve _any_ problem at all.
I'm generally skeptical on research and work in the field. However, the progress I've seen on GANs/deep fakes, natural language understanding, multi-modal, video understanding/activity recognition .. it is unbelievable how fast the field is moving. This may not result in new widgets next CES but may be game changing to society in a decade.
In academic research, GPT-3 and in general language models are applicable to a host of downstream tasks across different topics.
Edit .. reworded since it seemed insulting (unintentional).
> _This may not result in new widgets next CES but may be game changing to society in a decade._
And I am tired of hearing that ever since the 90s.
Clearly, deep fakes changed society. Made people more paranoid and less believing of what they see. Is that a positive change though?
Meanwhile, you still have to manually load dishes in the dishwasher, and robot vacuum cleaners stumble and block on the easiest of obstacles.
Call me a cynic but AI work needs to get out of its comfy well-funded bro-club corner and start to seriously try and solve real-actual-physical-world problems. It's all well and good that scientist X can model a text paragraph with this or that NN but when will he solve a car driving itself?
And don't get me wrong. I know science can take a _long_ time to make a breakthrough. It's not adhering to the same laws as everyday work -- I am aware and I am clamouring for science. But I don't feel the AI area is even heading in any direction at all. Could be wrong though.
There is a lot of work going on in many directions. Believe me, whenever the general public notices some "obvious" shortcomings or holes or missing research, there is usually already years of effort in the making and heaps of papers exploring it. It's just that there is a huge delay to media and popular consciousness.
If you just read the "AI" comment sections on HN it may seem everybody is just training a cat/dog classifier in Keras on ImageNet and nothing new is going on. In reality there's tons of work in one-shot/few-shot learning, unsupervised, self-supervised methods, combining modalities, quantifying uncertainty, robustness to attacks, etc.
Yes, robotics is still not at the stage where you could get a human-like dexterous dishwashing/T-shirt folding machine. But people are working on it as well.
However, not all companies need it. The "comfy well-funded bro-club corner" (why so much envy in the phrasing?) probably does not need physical robots, they just do data analysis, like if you need to process videos uploaded to Youtube, or you program an intelligent tool for photo or video editing software, that's also important and has nothing to do with robotic arms.
I think it's the fault of the media that "AI" appears as this single conceptual blur, when it's actually tons of different applications. It _is_ capable of a lot more than 1-2 decades ago. But it's not AGI, and not everyone needs to work on human-like/conversational AGI in a humanoid body.
Image analysis, machine translation, speech recognition, all these _work_ today at least to some extent and just a short time ago they just _did not_ work at all outside extremely carefully crafted cases in prototypes that fell apart immediately when the researcher wasn't there to keep it from collapsing (I mean this metaphorically: the predictions were extremely bad outside the scenario and dataset it was crafted for).
All of that is completely fair and I am not debating your well-made points.
My issue are a few things in particular:
- Recent evidence that most AI papers are approved by underground rings of peer reviews -- either on the principle of "you approve my paper, I'll approve yours" or because the authors know each other from university. That story was even posted here on HN several months ago.
- Any progress that the outside public sees is really small.
- Any non-small progress is never noticed by the general public. It's captured by a corporation because some deluded board member who never worked an hour in their life imagines that the company's small AI breakthrough will definitely hand them the keys to ruling the entire planet, I suppose. One example coming to mind: translation software. Why doin't we have that yet?! There are literal decades of research and some of the FAANG corporations bragged on their blogs, several times, that they almost solved it (Microsoft I think). All of the public-facing translation software packages are rather mediocre. Also, why don't we have publicly accessible and shareable NNs trained with millions of self-driving travels? Etc.
---
> _I think it's the fault of the media that "AI" appears as this single conceptual blur_
Many here on HN -- me included -- have the confidence to think they are not influenced by mainstream media on such topics. I personally got hyped for computers at 11-12 y/o partly due to the charmingly goofy but ambitious 90s sci-fi movies but now, almost 30 years later, I am not seeing that progress you speak about.
> _The "comfy well-funded bro-club corner" (why so much envy in the phrasing?)_
You deciphered it perfectly. It's envy, not going to deny the obvious. I am not making half-bad money at all but I have heard some insane numbers -- like $30k+ a month for being an AI researcher! -- and I in the meantime have to prove to my team leader that I am not completely useless even though I was recruited on the clear condition that most of what the company is working with is new to me and I'll need months to catch up. And I get times less salary. I admit I am envious of people who get to (a) do exploratory work and (b) be very handsomely rewarded for it.
I think even with that there's a good feedback that AI workers can extract from my rant: you guys are extremely privileged. Even more than a lot of programmers / sysadmins (who are already quite privileged compared to many other people).
> _It is capable of a lot more than 1-2 decades ago._
There's no ill intent behind the next phrase: I am honestly not seeing it anywhere. As a very pragmatic (and aging) programmer I'll believe it when my dishwasher can reach for the dishes and cutlery in the sink and load them by itself. I'll believe it when I can leave a robot at home while taking a walk with my wife and return to a spick-and-span-clean apartment.
Which brings me to...
> _If you just read the "AI" comment sections on HN it may seem everybody is just training a cat/dog classifier in Keras on ImageNet and nothing new is going on._
I am not denying that I do the same and I am likely judging this through not getting enough information. But my general intention isn't to be extremely well informed on what's happening in the area; it is more about asking the question "okay, fine, but WHEN will what you are working on be ACTUALLY useful, even a little bit, out there?".
I can see how many don't like having that question asked to them and so I guess people like me and an average AI researcher would never have a productive discussion. Damn shame. :(
I agree with you, but work is being done.
The problem is that AI isn't AGI. So a robot that can pick strawberries (exists, is in commercial use) cannot also pick apples (also a robot in commercial use). The apple robot needed its own R&D cycle. Grapes are different again, and olives, and lemons, and cherries, and almonds... And that's just horticulture.
In a highly diversified economy, it is going to take multiple decades and person-millenia of effort for the AI we have to show much impact.
Fair and understandable -- if you don't have Skynet then at least try and make the best with what you have. Cool.
It's just that in my eyes nobody is seriously trying to change the shape of the area. But I already admitted in a sibling (and much longer) comment that I am not well informed in the area -- and to be fair I don't aim to be.
I am just kind of getting disappointed that no real and tangible results are visible in society. What will the apple- or cherry-picking robots achieve save for leaving people without education jobless? Again? And they already struggle to find work anyway.
I am aware that most of us are slaves to the wage. But it's still saddening that nobody seems to take a stand and try and work on (a) something with a bigger and better social impact and (b) not laser-focused on short-term profit.
Having spent the last two months writing a novel, with GPT-3's help, I take issue with the claim that it's useless.
It's anything but useless. Sure, I do 80-85% of the work, but that remaining 15% is the portion that would otherwise have me stuck for days. In practice, it's a 10-50x improvement in writing speed.
Can you elaborate on how you use it? Do you have it generate next paragraphs when you're stuck, or something else?
I've elaborated significantly here:
https://forums.sufficientvelocity.com/threads/generic-pawn-t...
If that doesn't answer your questions, feel free to ask some in the thread. I don't check comment replies here very often.
(Note: The first ten-or-so updates were purely educational, and it's only the last few where I've been _trying_ for high quality text. If you just want to see how well that can work, then start at the end.)
What's your process for using GPT-3 to aid your writing?
See sibling comment.
I mean, don’t just take my word for it. Here’s Yann LeCun [1].
I will concede that I forgot about DeepFakes. How’d I do that? But still, cool now we can live in a dystopian future.
The point that I was trying to make still stands. Assuming that more data and more compute will solve all of our problems is a ridiculous position. The field might be moving incredibly fast, but most of the work I see is not great. Everyone is hyped. People claiming that we can replace all of statistics and machine learning with pure deep learning models forget about how hard it is to actually get these things working in the first place. Do you have 100 TPUs to run full blast for a week? I mean consider the series of Nature papers a few years ago. Google says, “I can predict earthquakes using NNs!” Somebody comes along and says, “I thought about the problem for a little bit and beat you with logistic regression.”
Self-driving cars have been just around the corner for the last 10 years. Just like with the rest of stats and ML, it’s not the place you have a lot of data that’ll kill you. It’s the edge cases.
[1]
https://m.facebook.com/yann.lecun/posts/10157253205637143?no...
Be careful, the narrative that you need tons of hardware to do anything is pushed by companies selling licenses to cloud APIs and cloud training platforms. In reality people routinely publish new results at AI conferences while only having access to a few GPUs. Not everything is about hardware. There are thousands and thousands of AI papers. Not all of them are about scaling up some model like GPT-3.
Yes, there is a lot of hype, a lot of misuse of ML by people who don't understand it. Yes, in some cases it makes no sense to use a deep neural net if logistic regression (which is a single-layer NN) will work. This is generally the case when your features are already very processed, curated and meaningful as in your earthquake example. But you will never solve e.g. image or speech understanding with logistic regression.
Perception on high-dimensional, complex, messy real-world signals is where deep nets excel. And they have unparalleled performance there, they blow everything out of the water. It's hard to overemphasize this. The last decade has turned many applications from does-not-work into works. A zero-one transition. Not 10% better, but it was _hopeless_ and now it is "quite nice". Still not divine perfection, but it's far from just being hype.
It can be hard for outsiders to appreciate this, but before 2010 or so, everything in computer vision was very fragile for real-world applicability. The algorithms only worked in one specific setting, like one particular assembly line, with all parameters hand tuned in the pipeline. You needed a lot of experience and intuition to configure your feature extraction pipeline and the regression method and it was _still_ very fragile. Much of research was only demonstrated on toy examples, synthetic data, or only on a few images. The shift to huge datasets is also something from the last 10-15 years. Before that people cherry-picked example, where their hand-crafted algorithm with hand-tuned parameters finally produced something resembling an okay result. I'm only exaggerating a little. These things really didn't work. Many of the concepts were already in place, but it still didn't work.
Could you list some conferences and papers one could use to catch up on ML and AI from zero?
Papers are meant for when you're already familiar with the basics. It also depends on your knowledge of prerequisites and goals.
But it's best to take courses and read textbooks. There are good free courses from reputable universities out there. I'd recommend starting with some math and CompSci foundations. The internet is full of recommendations because everyone and their mom is asking the same question. How to learn AI from zero. Most of those people then don't stick with it of course because it takes time and effort.
The other way is to just read "easy" tutorials and "instant" courses without any foundation, learn just the buzzwords and then be totally confused about what is actually going on.
Overpromising? Underpromising, if anything. I cannot understand the point of view that denies the _overwhelming_ progress that has taken place in AI. And to deny GPT-3’s mega advance — we now have a system that can achieve state of the art or near state of the art requiring a negligible number of training cases (i.e., few shot learning) — seems wrong to me.
If a cyborg were to "do everything" on deep learning would it have a meaningful model of reality or would it simply be behaving as if it did?
In particular, some recent work at Google has shown that you can do fine motor control and combine that with language, so that you can open a drawer and take out a block, and the system can tell you in natural language what it’s doing.
Does anyone know which paper that was?
What do you believe to be your most contrarian view on the future of AI?
Well, my problem is I have these contrarian views and then five years later, they’re mainstream.
Has every single one of his contrarian views panned out? What an arrogant quote. This just diminished him in my eyes.
Relax. It's an expected wisecrack at a question that is frankly not of very high quality.
I basically buy that DL will be effective, but I think the real innovation will have to be power efficiency e.g. how many GPUs and GBs of ram does GPT-_x_ need?
I wonder what humans will learn about themselves as we create synthetic competitors.
That perhaps it was already done before...
That enabling returns on capital entirely without labour functionally breaks capitalist democracies, probably.
Probably not - absolute standards of living will get even higher, but there will always be things only some people have, and more people want.
(This applies with or without synthetic life doing stuff for us.)
A new twist on the Terminator series. Skynet starts a war because the humans tries something which would influence the stock market.
Come to think of it, not much different to current reality... we already do that to ourselves.
Sure, if "everything" includes making humanly mistakes as well!
Hire a human that can only work 8 hours a day, or run and AI that can work 24 (and probably costs less). If their miss rate per hour is the same, definitely go with the AI.
Will the machine recognize them as mistakes? If so then its bootstrap time.
Since the fiasco of the nanotechnology, I became a very skeptical person when it comes to "disrupting" new technologies.
Hinton wants a Nobel prize awarded while his name is still in the ring!8-)) But that will likely happen a generation or two after Hinton is gone.
A Nobel for what? They don't do psychology, math, AI, statistics etc. And he already won the Turing Award.
I am very skeptical of the current approaches (supervised learning) making the quantum leaps being promised. This needs a paradigm shift of weakly-supervised learning and fine-tuning for specific tasks (a la human learning).
It is indeed where a lot of the current research is heading to if you count self-supervision as part of weak-supervision. Self-supervised learning brought massive improvement in NLP and is bringing state of the art result in vision after some time where purely supervised learning showed better result.
Weakly-supervised and self-supervised methods are extremely mainstream today. There is a flood of papers on these topics. Conferences are _full_ of them.
This comment was perhaps insightful 5-6 years ago, today it's absolute orthodoxy.
The thing is though, supervised just works way better. It's not that people don't want to do weakly-supervised, but that it's very hard!
My point exactly. Weakly/self-supervised learning is "very hard" because it's still an active area of research and breakthroughs are needed. I feel those a pre-requisite before we can credibly approach the promise of "doing everything"